But then again, the playback was done using JavaScript only and calculated in realtime.

Audiotool is a complex piece of software so you might ask how one could torture themselves by implementing it in JavaScript? We didn’t. Instead we started building our own vision of a cross-platform application framework a couple of months ago.

Introducing project Hiddenwood.

Hiddenwood is a collection of libraries and tools specifically designed to support different devices and platforms. The core libraries are the “driver layer” and always platform-specific with a platform-independent interface.
On top of that we provide a basic layer of libraries like our UI system, animation framework or managed collections which guarantee 0% garbage collection activity and have been battle-tested in Audiotool.

The framework is all about speed and consistency. The rendering pipeline is optimized for OpenGL and although we offer something similar to Flash’s display list a lot of features are not available because they would compromise the speed.

Speaking about speed: we are always interested in staying as native as possible on our target platform. So for the browser we emit JavaScript, for Android you will get the full DalvikVM performance and for the desktop you will get JVM performance. This approach has also another very important aspect. If you want to go platform-specific for certain features you can do that.
For instance if we want to render Audiotool songs on the server using a fork-join pool for our audio calculation this is possible and might not make sense on an Android device.

You write Java code and the supported platforms are native desktop applications, Android (minimum requirements are Gingerbread and OpenGL ES 2.0) and modern browsers. Now for browsers we even go one step further and support multiple options. That means if WebGL is not available we simply fallback to a normal canvas based render-engine. The same applies to some of the Android drivers.

iOS is of course important as well and we are actively researching the best option that will give us the most flexibility and performance.

We are currently working on two real applications built with Hiddenwood. So far it is a real pleasure to enjoy quick build times and simply test what you want on the desktop with great debugging capabilities. When you are ready you can try the same app on Android or in the browser — which might take a little bit longer to compile.

Because we see Hiddenwood as an application framework there are a lot of goodies built-in like a sprite-sheet based class generator. Think Image mixerBackground = Textures.mixer.background(); where mixer was the folder name and background the name of the file.

We believe that as a developer you really do not care about what kind of technology you are using and just want a great result. We also think that you should be able to reuse platform-independent code across multiple projects. However we do not want to take power away from the developer because if you know what you are doing: go for it.

Of course we are not the only ones with this idea. Nicolas Cannasse saw the signs years ago and invented haXe which gives you a comparable experience and Google released playN a couple of weeks ago which takes a similar approach (and requires another 25 committers :P).

But when we started Hiddenwood we wanted the Java tooling experience and playN was not public at that time. We also think that a game engine is not what you want to use for all kinds of applications. So we like to be able to give people the freedom to build their own game engine on top of Hiddenwood — and calculate physics in a different thread peut-être.
Speaking about threading: the only possible solution that works across all platforms is a shared-nothing architecture which we put in place. However if you write platform specific code you can use of course everything the platform offers and a lot of the Hiddenwood core libraries like the network- or cache-layer make use of multiple threads.

In the end what makes Hiddenwood special in my opinion is that we do not believe in write once run anywhere because that just does not make sense. The essence and philosophy behind Hiddenwood is to write platform-agnostic code using kickass-libraries and being able to reuse that. Audiotool on a tablet would look completely different from Audiotool running in a browser. And Audiotool on iOS would probably be also a little bit different from Audiotool on an Android device because there are simply different paradigms you should respect.

I hope that we can share more information with you soon. With the news of mobile Flash Player being deprecated and the ongoing demand for cross-platform development we have exciting times ahead of us. I am also super excited about the (beautiful <3) applications which we are going to release in the not so distant future.

Wikipedia defines STM as:

In computer science, software transactional memory (STM) is a concurrency control mechanism analogous to database transactions for controlling access to shared memory in concurrent computing.

First of all when reading this you might think that an STM is not necessary since ActionScript does not support true concurrency and also the shared-nothing worker proposal will not allow you concurrent memory access. While this holds true you may not forget that concurrency becomes an issue when you have just access to a single core but on multiple machines.

A lesser known fact about Audiotool is also that we have a built-in system for real time collaboration since its beginning. Due to conceptional issues and other prioritized features we were never able to roll it out as a full feature but the technology base is there.

When it comes to multiplayer gaming and network latency you will always have to figure out a system which allows you to share a certain state across multiple machines. In some games the state is synchronized across a certain server. An example is an FPS like Quake3. The Audiotool is unlike Quake3 not a FPS and different conditions have to be met. And this is the reason why I have choosen to implement an STM with very optimistic concurrency control.

The basic idea of my system is to use atomic transactions with commit and revert operations. Those are handled by a transaction manager which performs the basic tasks of an STM. That means: execute all transactions until an error occurs, roll back and try again. Each transaction in the Audiotool contains a guard. The guard defines whether or not one should still execute the transaction.

Here is an example of why a guard is important and how all this stuff works. You have two Audiotool instances running that share the same state. We call them A and B. A and B are connected via P2P no an arbitrary network. This means exchanging messages introduces latency and we cannot guarantee that A is at the same state of B.

So what happens? t₀ is committed by A, serialized and sent to B who is committing the same transaction. The same applies to t₁. t₂ is initiated by B, serialized and sent to A. Everything works fine until B commits t₄. While t₄ is being transferred to A, A created t₅ and committed it. This means A never saw t₄ when it committed t₅ and B would continue with t₆ ignoring t₅. A will actually receive t₄ after he commited t₅ so the system will revert t₅, commit t₄, commit t₅ if the guard does not prevent it, and both continue at t₆.

To explain the guard: Imagine the t₅ transaction includes modifications on a device in the Audiotool that is deleted by t₄. In that case t₅ will never be executed.

So what does this imply? You can guess that each modification of the Audiotool state is in fact handled by a transaction and we carefully have to pick the guards. E.g. if an editor is referenced by a transaction it is part of the guard. However this introduces a second issue: references. Sharing references of objects across a network is a tricky task. I am using “boxes” to solve this issue. A box can hold a value and it is guaranteed that it will hold the same value at the same state across the network. A box can be serialized and will be shared between commit and revert states. This means if t_n creates an object which is referenced by t_n+1 both will use the same box. This also means that I can safely revert t_n+1 and then t_n and commit them without having to worry that both still reference the same object. And this even works across the network. And in fact we can now also identify whether or not one transaction would conflict with a different one which is a very important condition for the actual implementation.

Right now you might have figured out that a transaction in the Audiotool has the following properties:

• It is serializable so it can be transferred across the network.
• All object references must be stored via “boxes”.
• A guard is used to determine whether or not to commit a transaction.
• Transactions can be committed and reverted.

A lot of stuff one has to reason about. But although we do not support live collaboration at the moment we are using the STM. Why? The reason is quite simple. The STM gives us history for free. Since everything is serializable via a very lightweight protocol we can take the last 500 transactions for instance and store them as a compressed ByteArray in memory instead of each as a single object with all its references.

Another very nice option is testing. A serializable history means that one simply has to press a super secret shortcut to dump the whole history and send it to the responsible developer who can reason about every single step that happened which ultimately led to the error. In development mode I usually commit, revert and commit each transaction. Then I also serialize and deserialize them and do the whole thing again.

Since transactions are known by the system we can also write a fuzzer that executes lots of crazy stuff. Sort of Android’s monkey mode. And ultimately I personally hope that we will get to the point where we will enable live collaboration in the Audiotool of course.

The last feature I implemented is the automatic transformation of arbitrary transactions to a compound transaction. You can guess what this feature actually means for the user :). In fact it is very complex and I implemented it with the STM using only a couple lines of code (of course that is just half the story …).

If you are interested in writing your own multi user application I think this is a very nice approach since collisions do not happen very often from my experience and the optimistic concurrency in a P2P environment achieves great results in terms of overall latency and feeling. The Google Wave protocol and their operational transform is also an interesting read.

I have been trying to spread the word about Scala during various Flash conferences and always tried convincing my colleagues and friends to learn this great language. A blog post by Alan Shaw about his experience with Haskell inspired me to write about my journey.

Why Scala

I was very unhappy with ActionScript and Java last year. ActionScript is a good start but needs to adapt the features of the dropped ECMAScript 4 proposal to become interesting and competitive in my humble opinion. Java on the other hand is only interesting to me because of its virtual machine. The language itself is simple and has the features you need to get the job done.

I got very late into the Java game and missed the bumpy early years. However I was never earning money with Java since I was doing only open source.

Open source is a hobby. I like doing it. So why not use a language that is fun too? I finished the first version of Apparat in 2009 using Java. But the language became more and more annoying because of all the boilerplate code you have to write and the massive violation of the DRY mantra. A particular example are Java’s generics. I was dealing very often with graphs in Apparat. I had a parameterized graph class Graph<V, E> and of course sometimes the vertices and edges are parameterized too. I was tired of writing code like this over and over again.

final Graph<BasicBlockVertex<AbstractOp>, Edge<BasicBlockVertex<AbstractOp>>> graph = new BasicBlockGraph<BasicBlockVertex<AbstractOp>, Edge<BasicBlockVertex<AbstractOp>>>();

So I was searching for a new language that allows me to focus on ideas. I could now talk about languages like Lisp, Clojure, OCaml, Haskell and C#; but I will just explain why I did not use C# — a language that is superior to most other languages. If Scala would not exist, I would probably do more C# work.

I do not think that interoperability is an issue with C# since Mono is a great project that gives you Linux and OS X support. But I used C# already for Apparat’s predecessor AS3C. So why not learn something new? This is why I went for Scala . I thought I can learn more about programming if I start using it — and boy was I right: I am learning something new each day since then.

Learning Scala

Most people that see Scala for the first time are immediately irritated by the syntax they do not understand. Same happened for me. When I first saw Scala I just gave it five minutes and went on to something else. The second time I came back to Scala I spent more time on the examples and tried understanding them.

Here is the problem: learning Scala is hard. My main issue was not the syntax but the fact that most people simply slap you with a lot of academic topics right in the face. As an autodidact you have to read a lot before you will write your first line of code if you follow that path. If you studied computer science and took a Haskell or Lisp course you will probably know about algebraic data types and monads already.

Instead I started ignoring most of the cool features and tried understanding the syntax first by writing actual code. You will find a lot of high quality blog posts on Scala very easy. When it comes to understanding Scala from an ActionScript perspective it is a little bit easier since you know anonymous functions already. Let us demistify a little bit of the syntax.

Take this ActionScript example.

const square: Function = function(x: int): int { return x * x }

You can remove nearly all type annotations because of Scala’s type inference. You still have to annotate method arguments. The compiler would complain if you leave those types out. This is a trade-off for readable error messages and compiler speed.

const square = function(x: int) { return x * x }

Next you can get rid of the return keyword in Scala. Because it is a functional language every usual statement is an expression. val x = if(true) 1 else 2 has a value for instance and is equivalent to var x: int = true ? 1 : 2. What does val mean? It is the const keyword in Scala. I prefer it this way. You have variables and values.

val square = function(x: int) { x * x }

This looks already much better in my opinion. And here comes the catch. Scala is a functional language. It is also an object oriented language but more on that later. But it borrows a lot of the nice features from other languages like OCaml or Haskell.

Because you are working with functions so often there are multiple short-hand notations available. But the key is to drop the function keyword and replace it with the shorter =>. Last but not least we do not even need the curly braces in Scala and our example finally becomes

val square = (x: Int) => x * x

You will figure out that => is used more often. This is another catch. Two particular things are confusing for the beginner. => and _. => is used in multiple places like definition of anonymous functions, types, scope definitions and self type annotations. I will probably not talk about the other two in this post.

It is worth noting that Int => Int is the type of a function that takes one parameter typed Int and returns a result typed Int.

So Imagine you have this definition:

val applyIntFunction = (x: Int, f: Int => Int) => f(x)

This anonymous function takes two parameters. One is typed Int and another one is a function. So we could write applyIntFunction(2, square). But what if we want to increase x just by one? One option would be applyIntFunction(2, x => x + 1). You can see here that Scala knows the type of my anonymous function already and I am allowed to omit it. This is not too bad. But since it is so common in functional programming to work with functions, an even shorter syntax exists. Meet the underscore.

applyIntFunction(2, _ + 1)

This is where it gets confusing at first. However you will love this feature and it is not so hard to understand. The Scala compiler expects the type Int => Int as the second parameter. _ + 1 is in fact an anonymous method that takes one parameter. But we do not know anything about the type if we look at the method alone. It is always important to think about the context. _ * _ is a method that takes two parameters for instance. This would not work in our example. What does _ * _ do? Probably multiply two things. But again it is a question of the context and the required types.


val applyTwoIntFunction = (x: Int, y: Int, f: (Int, Int) => Int) => f(x, y)
applyTwoIntFunction(2, 2, _ * _)


One of my main problems with Scala was the usage of the _ character and how it works. Fortunately a great post I want to highlight again already exists. It explains very well which purpose the underscore fulfills in which context.

So even if you think that Scala is hard to understand at first keep in mind that it is just a different dialect of expressing your ideas. If you have less to write you can do that not only faster but more concise. Once you start thinking in Scala it becomes of course absolutely clear that _ * _ means multiply those two things whatever they are.

A Language For Hackers

Scala is a great language for hackers. Four features make it very easy to write any library that feels like real code. Namely anonymous functions, the ability to omit braces and dots, type parameters and implicit conversions. Okay, what does it mean? I already talked about anonymous functions but omitting braces and dots? In Scala the code 1.plus(2) is identical to 1 plus 2. Whenever you have a method with exactly one parameter and an object on the left hand side you are allowed to omit both braces and the dot. First of all this is quite cool but also I did the mistake to make extensive use of this feature. In the beginning of Apparat I was omitting nearly all dots just because I can. It does not make your code more readable and I inserted dots and braces back into the code.

So when does it help? When it makes sense. This sounds weird but lets think about another example. Lists are a key element of Scala and any other functional language. "Hello World".toList returns a list of characters. If we want to have all of them as lower case we would map that list. Even ActionScript has a predefined method for this task, called Array.map. Noone uses it. The explanation is that it is simply to complicated. Imagine we have a List[Char] in Scala or an Array in ActionScript and we want to convert it to an Array where all characters are lower case.

This is the ActionScript version:

array.map(function(x: String): String { return x.toLowerCase() }).

This is the Scala version:

list.map(x => x.toLowerCase())

But lets keep in mind what we have learned and simplify it a little bit:

list.map(_.toLowerCase())

Remember what I said about omitting dots and braces?

list map { _.toLowerCase }

Et voilà. You can decide which one is better to read. The Scala version is short and nice. However if I would simply give you this snippet of code you do not know what it does as long as you do not know about the context. But we are dealing with language and context all day. Words have different meaning in a different context and it is not hard to guess that toLowerCase probably converts a character to its lower case representation.

Please note two more things. I left out the () after toLowerCase because it is a method without side effects. This is the case whenever your method does not modify any state in your system. Those methods are usually written in ActionScript like public function get toLowerCase(): String { ... }. And we omit the () as well.

One more example to convince you why this makes sense: Imagine we want to get rid of all the space characters in our now lower-case List (or Array in ActionScript).

array.map(function(x: String): String { return x.toLowerCase() }).filter(function(x: String): Boolean { return x != ' '})

vs.

list map { _.toLowerCase } filterNot { _ == ' ' }

This also explains why Scala is a great language for hackers. You can bend the language to your needs. Usually those hacks are categorized as domain specific languages. A language for a specific task. I have written a CSS DSL using Scala for instance.

In order to do such things you will need implicit conversions. A very cool thing that is out of scope for this article (Update: I wrote a post on implicits).

Conclusion

Scala can do much more than you might expect. I just covered a small part of it. For me it is important to explain why I like it so much and why I believe that it is an elegant language that makes programming fun again.

After one year of extensive Scala use I do not regret a single minute spending confused in front of my screen. First of all I like that someone spent a good amount of time thinking about the language. It is simply great that you can distinguish between a class, something you want to instantiate multiple times, and an object, Scala’s notion of a singleton, at compile time. No need for private constructors and the other boilerplate fuzz.

The Scala community is filled with very intelligent and academic people who are not arrogant and always treat you with respect.

Speaking for myself I can only agree that Scala makes you more productive and it was a great decision to start learning it. Scala is also not a pure functional language. In my opinion a great decision although not everyone will agree. But you can always decide whether or not you want to write a loop using a simple while or to go for tail recursive functions with immutable values. It is up to you.

I started to use Scala also at work as well. It is very nice for smaller projects like determining the similarity between users and storing that in a database. Currently I am the only one writing Scala in our office so most projects are still either Java or ActionScript based.

If you are into speedcoding: Scala is your weapon of choice. The one time I was using it on stage I was finished after about 75% of the time which is great if you target ten minutes but then my IDE crashed. Tooling is a different topic but has greatly improved in the last couple of months. If you want to learn Scala why not simply start with an old experiment you wrote in Processing and convert that to Scala? Since Processing is written in Java you can call its methods from Scala without any problem. I did not mention it yet but since both languages run on the JVM you have full interoperability between them.

Apparat is also a great example of marrying Scala with Java. JITB is for instance written in Java and Scala. I have choosen to go for Java because I am working with Java Bytecode in Apparat and at that point the Scala compiler was not very stable regarding the bytecode it emitted. This does not mean it was broken but it changed during versions which would have made my job much harder.

However even if you will not use Scala on a daily basis it will make you a much better programmer. This is also true for any other functional language. I used immutable lists in my as3-signals fork to simplify the whole dispatch logic and was able to delete a lot of unnecessary code.

Regarding Apparat, in the end I got rid of writing

final Graph<BasicBlockVertex<AbstractOp>, Edge<BasicBlockVertex<AbstractOp>>> graph = new BasicBlockGraph<BasicBlockVertex<AbstractOp>, Edge<BasicBlockVertex<AbstractOp>>>();

Which became

val graph = BasicBlockGraph.empty[BasicBlockVertex[AbstractOp], Edge[BasicBlockVertex[AbstractOp]]]

Oh wait, it did not. In fact it is now

def createGraph() = BasicBlockGraph.empty[BasicBlockVertex[AbstractOp], Edge[BasicBlockVertex[AbstractOp]]]
...
val graph = createGraph()

What does def mean? Go figure it out and enojy learning Scala. You will not regret it!

This year I had the honor to be part of the opener show for Adobe MAX 2010. I really appreciate that the people at Adobe working on the keynote took the risk to try this out.

Mike Chambers asked me already in April this year if I would like to do it. He had the idea of Erik Natzke generating art, me doing live coding and a DJ playing music. Eric Clark has been a fantastic supporter of Audiotool so I thought it would be really nice to have him do it. Luckily he agreed.

We had several meetings and finalized the idea of the performance. We really wanted to connect the work from everyone of us and create one piece. Otherwise it would have been only three persons doing their stuff. But eventually we got it to a point were I was visualizing the music Eric Clark was playing using pictures that Erik Natzke has been drawing on stage.

I as a developer always want to know how things work. So I also want to explain the technological aspect and challenge of such a performance.

Eric D. Clark was using Audiotool on a touch screen and we really wanted to show how Audiotool can shine there. The idea was of course to support gestures one might intuitively expect. This was done in Germany already. We are using a custom built event handling system so I had to add gesture support there. We do currently support only zooming the desktop since the schedule was so tight. In fact the week before going to MAX we also launched a collaboration of Audiotool and Coca-Cola. Everything was a little bit crazy.

However not as crazy as it went when we came to LA. When doing livecoding I usually prepare my final result already so I know what I want to create. You know where the little problems are and you can make the most out of the time you have. But because of the launch I did not have that much time to prepare. So I wrote one demo for me already in Germany. It was also my first time using the new 3D capabilities I was allowed to use which will be available as a public beta in 2011. Building software with a beta version of something is always fun because it can break for unknown reasons etc.

I also wrote a server that streams arbitrary recording devices into Flash at a very low latency. This was very important of course since I needed to access what Eric Clark was outputting. I did a version that does FFT and normal amplitudes but went only with the amplitudes since an FFT terrain looks not as interesting. It can. But then you have to do more analysis.
On the flight to LA I build another program that would help me save some time during the performance. It is basically automatically compiling and launching a Flash Player. A lot of people asked me about this actually. But you can built it for yourself very easy or I can even release the source code. It is really not much to do. I also contributed this code to Flexmojos as the continuous compilation goal. In fact this was my first idea. But since I did not have internet access most of the time and I was developing this on my tiny netbook I thought I want to have something as slik as possible.
Then there was another server involved I quickly wrote for Erik Natzkes machine. That one was also nothing special. It simply watches a folder and serves always the latest file. That way Erik only had to save an image to a certain folder with AIR and I would get it.
Last but not least we also used FlashMate. My old companion when it comes to multi monitor development. FlashMate hooks into the window procedure of the Flash Player and simply repositions the window always where you left it.

Because of the tools I built I never had to worry about starting the compiler, moving the Flash Player window around etc.

The program I wrote was actually quite simple. It was a subdivided plane in 3D space and I was drawing the audio feed I got from Eric Clark onto a BitmapData. That BitmapData was used as a heightmap for the vertices of the plane. In fact each pixel was assigned to exactly one vertex. That way I could get a really nice and smooth terrain. Thanks to Molehill this was rendering even at 1920×1080 without any hickups.
The heightmap was also used as a lightmap to fake some kind of ambient occlusion. I know that this was rather cheap. But keep in mind that we also had a limited amount of time. Even worse: We had some cue points in the performance. Like when Serge Jespers came on stage and was giving me the audio cable. At that point everyone was expecting me to be ready to visualize the amplitudes. Otherwise the whole thing would not have made any sense.

However we also wanted to show even more at the end because it would have been boring to see only the terrain flight the whole time. That is the reason why I flipped the terrain and started to rotate it. We agreed to this on Saturday, two days before the keynote. But luckily this is a rather cheap copy and paste operation. You take the code you already have and rotate the terrain by 180° around the z-axis. The rest of the animation was really only rotating everything around the y- and z-axis.
During the keynote we were also on IFB and at one point we were told that we have to stretch for five more minutes. That is the reason why the terrain was bouncing, freezing and whatever at the end.

I really enjoyed this whole experience. I love to have a look behind the scenes and it was an absolut pleasure to work together with everyone at the keynote. Friendly people all over the place. And a lot of hackers too.
However it was stressful. It is really not easy to type and concentrate in front of such a big audience. And it is not like a Flash event where you know most people anyways. All of a sudden there is the international press and all the executives sitting front row.

But I really liked the whole thing. Especially the last five minutes freestyle coding. I also want to thank Erik and Eric. The artwork matched the music very well. And I love the idea of us going on stage, each with a blank canvas on its own, creating something in a limited amount of time and having it connect as one piece.

Also worth mentioning: JITB ships now by default with Apparat. What does it mean? Quadruple-rainbow in your face. You can now run your SWF files with JITB if you follow the five golden steps to pure happiness:

1. Download and install Scala 2.8.0
2. Type scala in a terminal, then type println(System getProperty "java.library.path" split java.io.File.pathSeparatorChar mkString "\n") and remember one of the folders
3. Download LWJGL 2.5 and extract somewhere
4. Copy the native libraries which are located in lwjgl-2.5/native/YOUR OS/ into one of the folders that the Scala command showed you
5. Open a terminal, go into the Apparat installation directory and type ./jitb some.swf to launch JITB

Now here is the problem. JITB is not complete and so it will probably fail with your SWF — In fact I can nearly gaurantee you that it will not launch your SWF. I included JITB basically because it already allows you to use GLSL shaders very easy and I know a lot of people would like to play around with that.

However I have created an archive of all files I showed at FOTB besides the files that came from David’s laptop since I do not have them on my machine. You might be interested in checking out Example12.as which is the one that uses GLSL shaders. If you want to compile your own version you will need to link apparat-ersatz.swc as an external library since it contains the ContinuousFileLoader and GLSLShader class. Happy hacking!

• Download the JITB Examples
• Apparat on GoogleCode

Update: Also read Ray tracing with Pixel Bender using JITB by David Laenarts for details on the FOTB demos.

A part of the Flash Player is the ActionScript Virtual Machine (AVM) which is responsible for executing the code that is bundeled in your SWF file. This piece of software called Tamarin is open source and released under the MPL/GPL/LGPL Triple License.

Being able to look at the heart of the Flash Player does reveal a lot to the interesting reader. However such a virtual machine is a complex piece of software and you can be lost without any guidance and will maybe not even be able to compile it unless you are familiar with such systems. Jean-Philippe Auclair’s blog contains a lot of interesting Tamarin and Flash Player releated articles if you want to know more about the VM. Of course you can also read about MMgc — the garbage collection — directly at the Mozilla developer network.

This article is titled “Opening The Blackbox”. Although Tamarin might be some kind of blackbox to most of us the source code is available. We can dig into it and understand how it works. The Flash Player however is not completely open source and we do not know about its implementation details. Unless Adobe is opening up the source code we will not know which exact algorithm is used for BitmapData.noise for instance.

Fills an image with pixels representing random noise.

This quote taken from the documentation does not tell you much about how to calculate this noise. In this situation I would create some algorithm that fills a BitmapData also with random noise. But until we know which algorithm is used to calculate that noise it will never be the exact same noise as the one of the Flash Player. For now we will have to live with it. There is no (legal) way to figure out the algorithm.
Of course there are also other classes like ByteArray which have methods like readByte that do not need any extra explanation. But ByteArray also has some implementation specific details. How does it grow when you exceed its length for instance? Fortunately ByteArray is something you can find in the Tamarin source code. JITB uses a different algorithm however which you can find here.

But what else do we got? There are some funny things you will notice. Usually Sprite(x) is an explicit cast. This is true for any statement like this. However Number(x) is a conversion with specific details how a value is converted. This is also true for some other expressions like Boolean().

The playerglobal.swc contains all classes the Flash Player knows about. It should and it does with some minor differences. Of course it could be a bug on my end but some interfaces are missing. And they are suspicious since they cause/have special behaviour. Have a look at the AbcFrontend code. The Synthetic object contains part of the AST that is synthesized since those interfaces are missing in playerglobal.swc.

A lot of the Flash Player implementation details have made it into the DisplayObject and its subclasses. You cannot extend from DisplayObject and you cannot implement IBitmapDataDrawable. Even more interesting is the amount of classes that you cannot really use. flash.geom.Transform is such a class. How is concatenatedMatrix working if you would create your own instance of the Transform object?

Of course somewhere in the Flash Player will be some code that creates a Transform object. This will hold a reference to the original DisplayObject and concat the matrices up to the stage if you call concatenatedMatrix. The JITB implementation adds some special methods to the Transform class in such cases. Those methods are always prefixed with JITB$ if they are public. Another interesting fact is that you can modify the appearance of a DisplayObject by setting the matrix of the Transform object. This also has to change the scale, x, y and rotation values of the DisplayObject. There are a dozen of souch cases. Graphics would be an obvious candidate to talk about. I would advise you to take a look at Claus’ blog for everything you never wanted to know about shapes.

Also something you probably never cared about is the way how URLLoader works. I do not care about the specific implementation details either. However one thing is important to keep in mind. Loaders have to work asynchronous since they do not block. This means your Flash content runs while waiting for an event like Event.COMPLETE. This is where it gets interesting since threads enter the game. This would be a simple view: ActionScript runs in one thread, the loader in a second thread. First of all we have to keep in mind that the ActionScript code runs in a loop which is basically only triggered by events like Event.ENTER_FRAME or user input. Only the code in your DocumentClass is the one that is initiated without an event. So how does this look like in pseudo-code?

while(NoErrorOccurred && Running) {
  Dispatch Event.ENTER_FRAME to each DisplayObject
  Render DisplayList
  Dispatch Event.EXIT_FRAME to each DisplayObject
  Sleep(1000 / FrameRate - Elapsed)
}

Okay but how does an Event.COMPLETE make it into this loop when it comes from a different thread in which the loader runs? If the loader would simply dispatch this event when it completes the code would not run in the main thread. This means you could end up with ActionScript code running in parallel which would cause chaos and destruction. So what do you do? The current approach in JITB is to gather all events and dispatch them at the beginning of a frame. So basically the code looks now like this:

while(NoErrorOccurred && Running) {
  Dispatch All Gathered Events
  Dispatch Event.ENTER_FRAME to each DisplayObject
  Render DisplayList
  Dispatch Event.EXIT_FRAME to each DisplayObject
  Sleep(1000 / FrameRate - Elapsed)
}

The implementation specific details can be found in EventSystem.java class. There are also some utility methods like futureDispatch to make development easier. Dynamic audio requires to dispatch an event that is filled with content. The thread in which the audio is passed to the sound card has to wait until SampleDataEvent is filled with new content. This also reveals a problem. The audio calculation has to wait for a complete ActionScript frame until new material is available. In fact such synchronization points are problematic for a task that requires very low latency. You can hear this when your ActionScript code takes very long to process the new audio and you get a stutter-noise or audible gap. Unfortunately there is nothing you can do about this problem at the moment besides keeping the cost of your ActionScript code as low as possible.

Of course everything is only my interpretation. The Flash Player might not suffer from the problems I describe here since someone might have come up with a better implementation. In fact I would doubt that the Flash Player works for dynamic audio the same way I do it currently.

The last blog post of this series will be more about how you can implement parts of the Flash Player API in Java so that JITB understands them and vice versa. Of course we can create our own playerglobal.swc as well and add new native classes like GLSLShader which I demoed at FOTB.

ActionScript 3.0 is a dynamically typed language with function closures. Furthermore it has native XML support, scope-changes, implicit getters and setters etc. The main issue however is to convert the set of bytecodes ahead of time to statically typed Java code. JITB does not know anything about what is going on since it compiles all the code ahead of time.

JITB and TAAS

JITB makes use of TAAS. A three address code intermediate language. TAAS has its own set of rules and typesystem that I defined for myself. Method overloading is supported for instance. And there is no concept of array access. Code like x = a[0] is converted to x = a.getIndex(0). In TAAS terms the name of a method is also not worth a penny since it is always using a reference based lookup.

What I love about TAAS is that it has a really small instruction set. Array access is just a method call like any other method call. No extra instructions added. This makes optimizations really slick and easy to develop. Since the code is statically typed you also always know which method is being called and you could construct a call graph and do whole program optimizations which is a huge bonus. Escape analysis is something that is now finally possible for me.

JITB uses a frontend that parses ActionScript bytecode and creates an AST which is then transformed and passed to a backend that creates Java bytecode. The class and package structure is stored in a tree form. Method bodies are always stored in a graph representation. The interesting problem is to convert ActionScript into TAAS since it is no direct mapping and some statements are not supported in the same way you get them form the ActionScript bytecode. Then the next interesting problem is to convert this code into Java bytecode which is also not a direct mapping.

An example for such a problem are implicit getters and setters. TAAS does not have a concept of getters and setters but understands methods. So whenever the ActionScript bytecode has a GetProperty instruction I try to figure out if this is a field or not. If it is a field I generate a TLoad instruction, if it is an implicit getter it will be converted to a TCall instruction. However if it cannot be determined since the class might be dynamic a TCall instruction will be generated which uses the special TGetProperty method and this will be resolved at runtime.
When converting to Java a TLoad can be easily translated to a GETFIELD operation. When it is a TCall I first lookup whether or not it is a special native function. If it is a special function like TGetProperty I insert Java code like object.JITB$getProperty(name). If it is a normal function I simply insert an invoke operation.

Closures for Java

Java 1.6 does not support closures. This means Java does not understand the concept of a reference to a function. Code like addEventListener(Event.ENTER_FRAME, onEnterFrame) is not possible with Java since onEnterFrame is a reference to the onEnterFrame function which is defined somewhere else in your code.

I took the idea from Scala and how they support closures in the JVM. Basically when code like addEventListener(Event.ENTER_FRAME, onEnterFrame) occurrs it will be translated to the TAAS instruction TCall(TReg(0), TaasMethod(addEventListener, ...), List(TString("enterFrame"), TClosure(TaasMethod(onEnterFrame, ...))), None). This basically means that the method addEventListener is called on the object in register zero with two arguments of which one a closure and it has no result which is indicated by None.

This instruction is expanded to the following Java code. Please remember that it is always Java bytecode. I think it is just easier to understand if I write the corresponding Java code here.

this.addEventListener("enterFrame", new Function1() {
  override Object call(Event arg) { callVoid(arg); return null; }
  override void callVoid(Event arg) { onEnterFrame(arg); }
});

Function1 represents a function of arity one. The AVM can call any method as either void or with an expected result. If the method has a result and you call it via CallPropVoid the resulting object is ignored. If however you have a method that does not have a result and you call it via CallProperty the null object will be pushed on the stack.

The Java code in EventDispatcher which would call closures basically now only looks like this:

void dispatchEvent(Event event) {
  for(Function1 listener : listenersFor(event.type))
    listener.call(event)
}

Dynamic code in Java

Since I want to target Java 1.6 I do not make use of the Java 1.7 opcodes like invokedynamic. Because TAAS is already statically typed we have only in very rare cases the need to make a dynamic lookup. This is done by some helper classes or methods. Every ActionScript based class extends from jitb.lang.Object. The jitb.lang package represents the ActionScript’s toplevel. The Object class has certain helper methods like JITB$getProperty and JITB$setProperty. Those methods throw an error by default if an object is not dynamic but they can implement the behaviour to store values in a HashMap for dynamic classes.
However currently dynamic method lookup is not supported. Code like a['method'+Math.random()]() fails since it is impossible to know upfront which method is called. But this can be supported as well. It is just very cumbersome and annoying.

Emulating the AVM in the JVM

What kind of error do you expect for this kind of code: var x: String = null; trace(x.toString());? This creates a TypeError in ActionScript but a NullPointerException in Java. Same for things like var x: A = new A(); var y: B = B(x); which would result in a Java ClassCastException if A is not assignable to B.
JITB offers an option which is by default turned on the get the exact same exceptions like the AVM would throw them. This means that a Java cast is converted to a call like AVM.cast(x, B.class) with the option turned on or the simple Java CHECKCAST bytecode when the option is turned off. Basically expecting something to be null in a try-catch block is very silly in my honest opinion. So to get rid of this overhead you can disable this extra step of correctness.

Another interesting issue is the fact thatthe AVM can throw any object. Java however expects that a throwable object extends from the Throwable class. Of course the basic jitb.lang.Object class could simply extend Throwable. However I do not want that overhead and at any time. Also I do not want to break how classes extend each other. If you do something like throw "string" it is converted to throw new Throw("string"). Throw does extend Throwable and whenever it is catched the value is simply extracted.

To box or not to box?

ActionScript allows you do use int as an object with a lot of special behaviours. Same happens for values like Boolean or Number. var x: int; trace(x); outputs 0 but not null. But it is also possible to call 0.toString() which indicates that the integer value 0 is treated like an object that has methods like toString().
In Java however you have a real int primitive and the Integer object. 0.toString() does not compile but you would rather do Integer.toString(0).

Of course Java is much faster when using primitive data types when appropriate. JITB does this since it handles primitive types like real primtives for Java. This means a method like function add(x: int, y: int): int { return x + y} is converted to int add(int x, int y) { return x + y }. However when you have a method that expects an Object type a primitive will be boxed into a compatible type.

Last but not least what happens for code like 0.toString() or other methods? A possibility would be to emit code like new jitb.lang.Number(0).toString() but that does not make sense in most of the cases. Instead those calls are treated specially with a call like jitb.lang.Number.JITB$toString(0) and we get rid of an extra allocation.

Conclusion

I hope you understand a little bit better now how JITB works internally and what the main gotchas are in such an implementaiton. The journey is not over since I have to add support for anonymous methods and although I talked about things like throw I still have to add support for try-catch. Also a lot of other basic features are still missing but I doubt that they would cause any problems.

The other side of the story is the Flash Player API inside of JITB which is maybe worth a blog post like this as well since it reveals a lot of other funny and interesting gotchas.

However it happened again. I write a tool that is not even fully released yet and there comes Adobe around the corner with something that must have been in development for a while. Enough time to tell us that we can decide if we want to continue working on it or not.

I have no problem with Adobe developing such products. In fact I second that they have a Flex QA team which develops tools like FlexPMD and the recent coverage plug-in. However the way this process happens really frustrates me. Why would I want to work on something that Adobe is already developing somewhere silently and release it without giving anyone notice? But they do not owe me an explanation for what they do. It would also be arrogant to say that I demand to know upfront.

I am just sad that there is no dialog at all. This happened three times now. I will not continue doing this.

That does not mean I will stop working on Apparat and JITB. Only no more enterprise releated open source software.

As you might know I have done a couple of different tools for PixelBender already.

When I switched to Linux a while ago I had to discover that there is no compiler for PixelBender available and also no PixelBender toolkit. Even PBDT does not work on Linux since it requires a compiler.

In the current state Apparat can read and write PBJ files. I already implemented the format two years ago for the outline view plug-in. But this time Nicolas Cannasse’s great work on the hxformat library was my reference.

I will add two optimizations specific for PBJ files. TAAS will not be used here since it is only an unnecessary overhead. You do not have to crack a nut with a sledgehammer. The PBJ format is already register based and has certain useful invariants.

To cleanup some mistakes of the PixelBender compiler I will add:

• Copy propagation
• Dead Code Elimination

The compiler is not bad. Compared to the ASC it is a beauty of technology that makes use of the LLVM. However I think there must be some mistake in the code since unnecessary registers are used (Note: I doubt this is LLVMs fault but more the way how methods like sampleNearest are bound by Adobe). This can be solved with the simple optimizations I will add.

I will add some more things as well like a GLSL conversion or a DSL to write PixelBender kernels in Scala. If we get lucky this will also lead to a true cross platform opensource compiler for the PixelBender language. I think I will name that compiler Hydra :P

This time recorded on Ubuntu. Did I mention 64bit already?
Do not miss my session at FOTB for a live demo.

I want to put a lot of things regarding JITB in perspective here as a followup to the various comments that I received via mail, reddit, etc.

Flash Replacement

JITB is not a Flash replacement. Imagine a F1 race car and a family van. The family van comes with air conditioning, a radio and a lot of comfort. For the F1 car you will probably have to wear a helmet and need an engineering team to get it running.
The important factor for me is that JITB will run very fast under certain conditions. A race car is also not a jeep. JITB will not support all features of the Flash Player. If you search for a Flash replacement you might want to take a look at lightspark.

Performance

Java is slow and the world is flat. That being said I will not talk about Java performance. JITB compiles to native Java code with a certain overhead. An ActionScript 3 closure is converted into an anonymous class for instance.
JITB converts ActionScript 3 bytecode to Java bytecode at runtime and performs various optimizations. This way we can leverage the speed of the JVM for ActionScript and get great results.
That being said there is also a problem of course. Startup time is not very good. JITB and its parent project Apparat are implemented in the Scala programming language and written for multi core architectures. The startup costs are high and JITB is only useful for long living applications. However compiling ActionScript to Java can be done AOT. This means you need to deal only with the normal JVM startup time. The Flash Player API is implemented in pure Java.

The compiler used for JITB is doing relatively naive optimizations at the moment. Namely constant folding, copy propagation, dead code elimination and strength reduction. An older proof of concept version I implemented a year ago featured also loop invariant code motion, inline expansion and tail-recursive optimizations. The new compiler framework is very powerful and I will add even more algorithms and a graph coloring register allocator for example. So there is some more NP-complete fun ahead making startup time even worse.

Purpose

We need a Flash Player that performs very fast for special purposes. We could use JITB at Audiotool to render mixdowns of tracks or other companies could use it to run their ActionScript code on the server side. You can use JITB to create an offline application that runs on a client machine. You can hack JITB to do what you want. Someone could potentially write a web framework for ActionScript and you could execute that code on Google App Engine if compiled AOT. And it is definitly possible to create Android applications using ActionScript. JITB’s terrain is everything but the browser in my opinion. However I can imagine that someone would be interested in creating a plugin version. With some clever simple caching algorithms it could make sense to get around startup costs but I am not interested in doing this.

Legal

There are no legal obligations to face since both the AVM2 and SWF specifications are published by Adobe. It is also not illegal to compile code for the JVM.

ActionScript Subset

By “a subset of the language” I really mean the language and not the API. JITB supports currently only statically typed ActionScript code. However the internal language model can be adjusted to support dynamic typing as well and with Java 7 things got even easier. In fact JITB does support dynamic code as long as it is able to infer types correct. Does this mean that I am willing to implement the full Flash Player API all by myself? Hell no.

In the next weeks I will probably create a better example that is easier to understand and get some other stuff ready so that you can try everything for yourself.

My talk at FITC San Francisco is over and I want to share some of the anouncements from today with you. At the end of my talk I was showing JITB.

What you see in the YouTube video I posted a while ago is a Java program executing a SWF. For FITC I added some more code and an OpenGL based Display List renderer. In other words: I wrote a Flash Player.

However I should rephrase that statement and say I am attempting to build a Flash Player. The current state is available in the sf2010-sprint clone of Apparat. I will merge the changes into the main Apparat branch when I am back home in Germany.

JITB is currently able to translate a subset of ActionScript code at runtime into Java bytecode and runs nearly at the same speed as native Java. This is a really huge improvement compared to standard ActionScript performance. A lot of smart people worked on the JVM and made it really fast. Apparat will allow you to leverage all this hard work in the future. I am also shooting for Java interoperability at some level so that you can call Java classes from within ActionScript. Hopefully you will be able to use JITB on your desktop machine, on a server or on an Android phone. Basically everywhere Java runs.

There are still a lot of things missing. The whole Flash API needs to be implemented. And the Display List rendering needs a proper OpenGL implementation. However I thought this might be some cool stuff to share with you in its early stages.
My hope is that more people start contributing to the project. Maybe some OpenGL guru wants to take care of the Display List rendering or someone else likes to help implement the Flash API in Java.

I also showed a Raytracer by Nico Zimmermann during my presentation and promised to put the URL on my blog so here it is. His company is called Britzpetermann and the address is http://www.britzpetermann.com/.

Update: Please do not think that this implementation is 30x faster than the Flash Player developed by Adobe. One(!) microbenchmark is never a number you should count on. I would like to make clear that I never said this.

A big thank you goes out to the organizers and the Flash community. It really shows that the opensource work I do is appreciated and not meaningless.

Part of the prize I already received has been donated to Eugene Zatepyakin. I think he is doing great work on ASSURF and was in desperate need of an FDT license ;)

Thank you. I hope I will be able to surprise you with some new stuff later this year.

Use the inline feature with care. Apparat does not try to optimize your code and performs nothing but dead simple inlining. This can lead to slower code due to the creation of lots of local registers. Your code gets also much bigger and will require more space in memory. I am actually not a fan of manual inlining at all. I think it makes only sense to inline code if you have a powerful optimizer available that will cleanup the whole mess.

The fun story about this example is that the inlined version is slower using the lastes Flash Player release candidate if you have only 40.000 particles. That is why I increased the number of particles to 80.000 ;). I developed the example using an old standalone player and the inlined version was nearly twice as fast. However when I watched the example in the browser with the latest release candidate the game was completely different. Kudos to Adobe for significantly improving the Flash Player performance!

• AS3-only Version
• Inline Version
• Source