You might ask what the hell this is about. So in the current state you can download Apparat from Google Code. Then you have to have a working Scala installation which has been used to build Apparat. Whenever Scala is updated, I have to updated Apparat, and you will have to download Apparat again and install a new Scala version. This is absolutely cumbersome. This will change of course when Scala 2.8 is officially released.

I know that some people will always require command line access for Apparat. I am sorry for you at the moment that we have to play this game until 2.8 is out.

For the rest this is good news. If you have a working Maven 3.x version you can use the Apparat plugin and get automatic updates. This means you have to configure your project only once and do not bother with Scala and Apparat anymore. In fact if you want to you could also use the Apparat snapshot repository and get live updates.

Project configuration and setup is something one should not have to deal with. Therefore you can use the archetypes. I have built one for TurboDieselSportInjection.

mvn archetype:generate \
  -DarchetypeRepository=http://oss.sonatype.org/content/repositories/snapshots \
  -DarchetypeGroupId=com.googlecode.apparat \
  -DarchetypeArtifactId=apparat-archetype-tdsi \
  -DarchetypeVersion=1.0-SNAPSHOT

This terminal command is all you need to do to create a new project that is ready to compile — with TurboDieselSportInjection enabled. And it will automatically use the latest Apparat version.

Use the inline feature with care. Apparat does not try to optimize your code and performs nothing but dead simple inlining. This can lead to slower code due to the creation of lots of local registers. Your code gets also much bigger and will require more space in memory. I am actually not a fan of manual inlining at all. I think it makes only sense to inline code if you have a powerful optimizer available that will cleanup the whole mess.

The fun story about this example is that the inlined version is slower using the lastes Flash Player release candidate if you have only 40.000 particles. That is why I increased the number of particles to 80.000 ;). I developed the example using an old standalone player and the inlined version was nearly twice as fast. However when I watched the example in the browser with the latest release candidate the game was completely different. Kudos to Adobe for significantly improving the Flash Player performance!

• AS3-only Version
• Inline Version
• Source

So if you want to have quick and dirty inlining capabilities: this is for you. It is an easy fix for a feature a lot of people have asked for. I will continue working on TAAS to implement this much better in the future.

However when deploying you can spend some extra time compressing your SWF files with the best tools available. 7-Zip is for instance such a tool and achieves a great compression ratio.

If you have the 7z executable on your PATH Apparat will make use of it to achieve the best compression for your SWF files. More configuration options are available here.

This feature is currently only implemented for SWF files and not SWCs. However every tool will make use of it by default. So if you are running Reducer and have no graphics in your SWF files you can still get a better compression.

An example is straight from the sources:

reducer -i as3\Apparat.Tests.AS3\bin\Test07.swf
[i] Apparat -- http://apparat.googlecode.com/
[i] Launching tool: Reducer
[i] Waiting for 7z ...
[i] Compression ratio: 18.224573%
[i] Total bytes: 310
[i] Completed in 547ms.

Note please that this SWF file does not contain any graphical assets and we got still a ratio of 18%. Our main audiotool.swf file is compressed by about 10% — about 200kb — and contains no graphical assets either.

You can download the latest Apparat on Google Code. Please note that you will need Scala 2.8 RC1 as well.

However the current version is named Firestarter for a reason. What you see online is just the tip of the iceberg. I can not wait to start implementing all the cool features that will separate us from traditional software ;)

An important aspect is that when I am talking about modules I am not talking about “Flex Modules” but the more generic term for a part of a project.

First of all you have to make sure to fulfill some requirements. You can not assume that every developer is working on the same platform using the same IDE with the same settings. But you must assume that your project will not be built from an IDE. You must also assure that the dependencies between projects are up to date across different platforms and IDEs. Once you have figured out how to do that you might want to think about a strategy to compile your modules.

Today I want to explore the idea of parallel compilation across a cluster of machines to yield optimal compile speed for ActionScript projects. First of all, you can not assume that a developer can always keep all modules up to date. Depending on the dependency graph of your projects compilation migh take very long. For instance we have 103 modules at Hobnox. This means we have 103 different compile targets with different dependencies, each yielding its own SWF/SWC file. We also have a “standard library” that is included in nearly every project. Changing this library will require the developer to recompile all 103 modules to update them.

To illustrate this example I want to explain what we went through.

1. We used FDT to develop our project. Since FDT is Eclipse based every module was represented as a separate project with dependencies to other projects. FDT however compiles your project only once you launch it. And it does not do this automatically for your dependency graph. As a result you have to manually compile 103 modules in the correct order. This would not be the problem if FDT’s live error highlightning and code analysis would not work on a SWC basis. But since the actual source-code between projects is not analyzed we had to launch all 103 modules to propagate a possible error or change. This was for us the point when we had to make a switch.
2. Ant was our second idea to build our FDT projects automatically. However we had to keep module dependencies up to date in all the different Ant files because the FCSH requires that you include external libraries of a library also in a successor of another library. This means if A requires B and C requires A. Then C requires also B. So we had to keep 103 Ant targets up to date for each change.
3. A custom build. We started working with a custom build tool I have developed in Scala. This build tool was smart enough to figure out which modules have to be recompiled without launching the ActionScript compiler. It also figured out which projects could be compiled in parallel since the ActionScript compiler is written in a way that IO is not the only bottleneck. We had an enormous improvement in compile time and usability for the developer since module dependencies where handled automatically.
4. We switched from FDT to IntelliJ IDEA since it offered us more features that we needed. IntelliJ came also with a much better strategy to compile modules. Basically it had the feature we always missed for FDT to automatically include libraries and to compile modules in the correct order automatically. However the guys at JetBrains made a mistake by counting on FCSH. The short story is that FCSH is absolutely not capable of handling a large codebase. What happened to us was that when we started a build FCSH would take up as much memory as it could, getting much slower until it finally crashed. Then FCSH started from scratch, consumed the maximum amount of memory again and crashed. This happend until a project was finally compiled but took also more than 15 minutes for a single change we made. This was of course unacceptable. Fortunately JetBrains implemented a feature request from us into IntelliJ IDEA which comes close to our custom build. From now on projects are compiled in parallel without using FCSH and without asking launching the compiler if no change has been made. This way we are able to work again. However only if we compile just a short amount of all modules since the main bottleneck is still the compiler.
5. A continuous integration server was our last option. This is a custom build server and we are using it until today. After each SVN commit we are building the required modules and their dependencies now automatically on a dedicated server. A change in the standard library takes about four minutes to complete now on a quad core. But the good thing is that we have set our main project to exclude a lot of libraries. The custom build takes care of that and we are loading now modules from the server. That way we can still compile local and include libraries we frequently change. Other libraries are loaded from the server and included at runtime.

So the question is if this is now the ultimate solution. A custom continuous integration server that takes more than 4 minutes to propagate a change after a SVN commit? I do not think so. And here comes Scala into the game again. There is a project available called Swarm. It makes use of Scala’s support for serializable delimited continuations. This means you can pause the state of your current program, send it to a different node in your cluster and continue with the program on that machine. And Swarm makes this task extremely simple. So for now a new research project is to take advantage of this feature. We can basically have one master server like Hudson or TeamCity and write a plugin for it that makes use of Scala Swarm. Swarm could be based on nodes in the office, normal computers and maybe a couple of dedicated servers to compile modules in parallel on multiple nodes. Each node would have to keep the VCS up to date. That way we should be able to compile projects much quicker and have a much better workflow in a large codebase. Besides that if you work with a CI server than you could also have the task of code analysis using FlexPMD for instance done on another machine as well.

I am really looking forward to this project when I have some time to work on it. My goal is to develop a plug-in for TeamCity that we will use internally first. I expect to reduce the time for a full rebuild from about 4min to something like 30sec. This would be a huge benefit. I also hope that we will be able to make that plug-in available to the public.

The Java to SWF compiler does not compile Java sourcecode but JVM bytecode to ActionScript bytecode. This means I do not have to teach my program the Java language. It only understands JVM bytecode. This seems like an akward decision on the one hand since working on the bytecode level implies lots of problems. But it turns out that this was a really cool decision on the other hand. “Java to SWF compiler” is maybe the wrong description. “any language that compiles to JVM bytecode to SWF compiler” is maybe better.

So what does any language mean? Here is a list of JVM languages. Now you feel maybe like I did after reading that tweet. And I am really looking forward to get Scala up and running.

Some problems still exist. Threading is one issue and I will basically have to do what Scott Peterson did for Alchemy. But reflections, annotations and method overloading have to be solved as well. Some glitches may exist even after figuring everything out. Stacktraces will look pretty weird. However I think this is a really cool project.

Update: I forgot to mention something important. Java supports native code. This means you can build a library that works with OpenGL for instance. Those native methods can not be converted. There are also some other things that do not work. File access is just one of them.

Of course doing something at home influences work. I decided to write a build tool for the Hobnox AudioTool which is about 200 lines of Scala code. Cool thing is that this tool replaces a manually maintained Ant build and all project dependencies are always correct. Plus analyzing the dependencies in a more powerful way allows me to spawn the compiler in parallel. Building the AudioTool is now twice as fast and much more comfortable.

When learning a new programming language you also learn about new concepts. Functional languages in general have a different approach to nullable types. I know Scala is not the only one but let me introduce the concept in terms of ActionScript.

When you have a method that returns either a result or nothing: what do you do? Imagine you have some kind of service and a Dictionary of users. Requesting a user works by his unique id. The Dictionary is private to the class since you want to keep it read only.

function getUser(id: String): User {
  return hasUser(id) ? users[id] : null;
}

If I would now simply ask the service for an unknown user and do something like if(getUser('xyz').isLoggedIn) { trace('Hooray'); } I could probably and up with a null-reference error. No one checks for me if the user exists. So what else could we do? Write a lot of boilerplate code and prepend a check if the user is null or not each time we request one from the service. A much better approach in my opinion is to throw an error as early as possible. In this case we would rewrite the method to something like this:

function getUser(id: String): User {
  if(hasUser(id)) return users[id];
  throw new NoSuchUserError(id);
}

In this scenario we get informed about the error as early as possible. But we are stuck again. First of all ActionScript does not enforce you to catch possible exceptions. This means if you do not read the documentation of a method carefully or look into the source code of a method before calling it somewhere you will never know that it throws an exception. And what if we are actually in a scenario where we do not expect errors for non-existing objects? Think of the Dictionary object throwing each time an error when you access it and the result is null. How could I even check if an object exists in a Dictionary?

try {
  dictionary[key];
  return true;
}
catch(noSuchElementError: NoSuchElementError) {
  return false;
}

I guess you see that this can not be the solution to our problem. In a real world example you may deal with your own collection of objects instead of a Dictionary of course. So we have to get rid of exceptions and null for the optimal solution. Scala’s approach to this problem is the Option type. We always abuse null as a placeholder when we want to express that an element does not exist. The Option means that either Some or None result exists. Rewriting our getUser function using this approach would yield the following ActionScript code.

function getUser(id: String): Option {
  return hasUser(id) ? new Some(user[id]) : new None();
}

Why is this much better than the old approach? When calling the method you will always know that the method has only an optional result value. We get rid of the exception and null values. Our only problem at the moment is ActionScript. The result is now untyped. In an ideal world this method would be written as:

function getUser(id: String): Option.<User> {
  return hasUser(id) ? new Some.<User>(user[id]) : new None.<User>();
}

However we can still tackle this issue by implementing null-representations of our objects. Imagine the User class. You could rewrite the code to something like this.

function getUser(id: String): IUser {
  return hasUser(id) ? user[id] : new NullUser();
}

final class NullUser implements IUser {
  public function get isLoggedIn(): Boolean { return false; }
  public function get name(): String { return 'null'; }
}

And even if you are interested in null-reference errors you could rewrite your code to something like this:

final class NullUser implements IUser {
  public function get isLoggedIn(): Boolean {
    CONFIG::ThrowNullReferenceErrors { throw new NullReferenceError(); }
    return false;
  }
  public function get name(): String {
    CONFIG:: ThrowNullReferenceErrors { throw new NullReferenceError(); }
    return 'null';
  }
}

It is definitely a very different approach. A functional language like Scala allows you to deal much better with Options. But it makes sense to diferentiate between an uninitialized variable which is null and an optional result of a function. Unfortunately this is at the moment very painful with the lack of generics in ActionScript.

Now if you add one and one together you see that the TAAS compiler can be used as a very strong decompiler. My own tests have shown that it will work flawlessly where other commercial decompilers output rubbish. Since the compiler behaves like the Flash Player it will “execute” the bytecode in order to parse it which means it has a very highlevel understanding of the structure inside the SWF.

The only question is now what to do with the source code. I wrote the decompiler for my session at FOTB to show much easier how the optimizations behave. It is also a great tool to debug errors. But should it be opened or not?

To take it one step further one might also be able to write an obfuscator using the TAAS compiler. In my opinion it would be cool to have a strong decompiler and obfuscator, both being open source. We might also add an option to protect SWFs from the decompiler by adding something to the SWF metadata for instance. Of course this is just a simple rule which could be removed by someone once the code is open. What do you think?

The recording of the jam session is also available. I just want to note that I used the processing library inside Eclipse for the speedcoding. This time I also practiced at home unlike for other jam sessions. The visualizer was an homage to a table designed by Daft Punk for habitat.

So how does this work? Because I am now working on the AudioTool backend for our highly anticipated launch I had to switch to fulltime Java development. In order to understand the JVM better I started to search for the optimizations that the HotSpot VM performs and stumbled upon a framework called Soot. Voilà. Reading about Soot made me question why nobody is working on such a framework for ActionScript. Probably because universities see Flash just as a toy. For us developers being dependent on the Flash Platform it is of course more than that.

I will quickly explain what Soot does. It reads compiled Java class files and converts the machine code they contain into a three address code representaiton. This code is then heavily optimized. TAAS is basically doing the same. I also want to note that Soot is by far more complex and complete than TAAS right now. It would be megalomania saying that TAAS is Soot for ActionScript. But you get the idea. The one framework converts Java to three address code and the other framework converts ActionScript to three address code. Maybe it is possible to connect them.

This is what I did for the Java compiler. The whole pipeline looks like this:

1. Compile Java sourcecode with the Java compiler of your choise
2. Use Soot to get the three address code representation of a compiled binary
3. Create TAAS expressions for the expressions that Soot uses
4. Connect the TAAS graph the same way the Soot graph is connected
5. Compile the TAAS graph to a SWF

This is about it. There is some glue-code involved here and there and I did not implement all Java expressions yet. I did also not bother about threads. Basically one could do what Scott has done for Alchemy. The cool thing is that we get highly optimized Java code because of the Java compiler and Soot. Then the TAAS compiler can run a second time after linking is done and perform platform specific optimizations like inlining and loop invariant code motion for certain expressions.

I already wrote a couple of classes and packed them in a SWC. Those mimic the behaviour of the Java classes like java.lang.System or java.util.LinkedList. On the Java side I implemented the Flash classes like flash.display.Sprite and flash.events.EventDispatcher. The interesting part is that all the methods those classes contain are marked native which means they have no implementation since they are native to the Flash Player.
This is also a great advantage. It is slow in Alchemy to call Flash methods and to communicate with Flash classes. The Java approach does not have this tradeoff. I also do not have to trigger the ActionScript compiler. The conversion form class to swf is entirely done using Soot and TAAS.

So far for the Java compiler. But in order to show a little bit more of the potential I wanted to have a C# compiler working as well. The great thing about advanced languages and a large community is that people have already built a lot of tools. It did not take too long to find an advanced framework that converts C# to Java. Actually this is not be the best approach. But I could not find the “Soot for C#” the evening before my FOTB session.

So I am using net2java to convert .NET code like C# or VisualBasic to Java. That code is then compiled using the Java compiler and I am using Soot again to convert the code to TAAS without much stress.

For me the most important part is to know that it is possible to compile .NET source code and Java binaries to SWF. To complete those tools one just has to implement all the missing expressions and the standard Java library in Flash. Everything could also work the other way around. Compiling from TAAS to .NET, Silverlight and Java is another option.

Before FOTB I was not sure if I can present Apparat at all. For me as an autodidact it is not always easy to solve certain issues. Mario Klingemann talked about Magic Moments in Brighton. I guess I had one after a long time of thinking and failing. I got really into the flow and everything became so clear. Stay tuned for more detailed posts on the technique behind the different compilers.

However this was a really special conference for me. The standing ovations made me completely happy and I was absolutely overwhelmed by the response. After all those sleepless nights it was a huge gift of appreciation for my work. I will definitely never forget that moment and all I can say is thank you. If you spend so much time working on an opensource project this is really the best that can happen. “Leaving the Sandbox” came from a long background. The end of the session was a really emotional for me. Thank you John for inviting me.

Strength reduction can already handle about 55 different cases at the moment. For instance it will convert code like if( x - 1 == 0 ) into if( x == 1 ). But it will also remove expressions that other optimizations could introduce. x + 0 is something that might occur during inline expansion. Or x + Number.NaN for example.

But I really like the tail recursive optimizations. TAAS can detect if a method needs to call itself or not. Take this code for example:

private function sum( i: int, value: int ): int
{
	if( i == 0 )
	{
		return value;
	}

	return sum( i - 1, value + i );
}

You see that the last statement sum( i - 1, value + i ) is tail recursive. This means we can replace the recursive call to sum() with a jump to the beginning of the method after changing the method parameters. The sum() method becomes converted to something like a simple while loop. This means also that one could have written code like this in the first place since this optimization applies only to methods that do not have to be recursive after all.

Knock yourself out with some examples:

• Before / After (Inline)
• Before / After (Inline)
• Before / After (Tail recursion)

Hopefully I will meet a lot of interesting people and Flash Player engineers. It will be my first MAX so I am really curious.