Author Archive for joa

Compiling ActionScript In The Enterprise

Published
by
joa
on Mar 6, 2010
in Flash
. 19 Comments

If you ask me if ActionScript is ready for the enterprise then this question is hard to answer. Every fairly huge codebase requires some maintenance. You will have to figure out a strategy to identify modules and separate them. You will also have to handle the dependencies between modules. Last but not least you have to compile those.

An important aspect is that when I am talking about modules I am not talking about “Flex Modules” but the more generic term for a part of a project.

First of all you have to make sure to fulfill some requirements. You can not assume that every developer is working on the same platform using the same IDE with the same settings. But you must assume that your project will not be built from an IDE. You must also assure that the dependencies between projects are up to date across different platforms and IDEs. Once you have figured out how to do that you might want to think about a strategy to compile your modules.

Today I want to explore the idea of parallel compilation across a cluster of machines to yield optimal compile speed for ActionScript projects. First of all, you can not assume that a developer can always keep all modules up to date. Depending on the dependency graph of your projects compilation migh take very long. For instance we have 103 modules at Hobnox. This means we have 103 different compile targets with different dependencies, each yielding its own SWF/SWC file. We also have a “standard library” that is included in nearly every project. Changing this library will require the developer to recompile all 103 modules to update them.

To illustrate this example I want to explain what we went through.

  1. We used FDT to develop our project. Since FDT is Eclipse based every module was represented as a separate project with dependencies to other projects. FDT however compiles your project only once you launch it. And it does not do this automatically for your dependency graph. As a result you have to manually compile 103 modules in the correct order. This would not be the problem if FDT’s live error highlightning and code analysis would not work on a SWC basis. But since the actual source-code between projects is not analyzed we had to launch all 103 modules to propagate a possible error or change. This was for us the point when we had to make a switch.
  2. Ant was our second idea to build our FDT projects automatically. However we had to keep module dependencies up to date in all the different Ant files because the FCSH requires that you include external libraries of a library also in a successor of another library. This means if A requires B and C requires A. Then C requires also B. So we had to keep 103 Ant targets up to date for each change.
  3. A custom build. We started working with a custom build tool I have developed in Scala. This build tool was smart enough to figure out which modules have to be recompiled without launching the ActionScript compiler. It also figured out which projects could be compiled in parallel since the ActionScript compiler is written in a way that IO is not the only bottleneck. We had an enormous improvement in compile time and usability for the developer since module dependencies where handled automatically.
  4. We switched from FDT to IntelliJ IDEA since it offered us more features that we needed. IntelliJ came also with a much better strategy to compile modules. Basically it had the feature we always missed for FDT to automatically include libraries and to compile modules in the correct order automatically. However the guys at JetBrains made a mistake by counting on FCSH. The short story is that FCSH is absolutely not capable of handling a large codebase. What happened to us was that when we started a build FCSH would take up as much memory as it could, getting much slower until it finally crashed. Then FCSH started from scratch, consumed the maximum amount of memory again and crashed. This happend until a project was finally compiled but took also more than 15 minutes for a single change we made. This was of course unacceptable. Fortunately JetBrains implemented a feature request from us into IntelliJ IDEA which comes close to our custom build. From now on projects are compiled in parallel without using FCSH and without asking launching the compiler if no change has been made. This way we are able to work again. However only if we compile just a short amount of all modules since the main bottleneck is still the compiler.
  5. A continuous integration server was our last option. This is a custom build server and we are using it until today. After each SVN commit we are building the required modules and their dependencies now automatically on a dedicated server. A change in the standard library takes about four minutes to complete now on a quad core. But the good thing is that we have set our main project to exclude a lot of libraries. The custom build takes care of that and we are loading now modules from the server. That way we can still compile local and include libraries we frequently change. Other libraries are loaded from the server and included at runtime.

So the question is if this is now the ultimate solution. A custom continuous integration server that takes more than 4 minutes to propagate a change after a SVN commit? I do not think so. And here comes Scala into the game again. There is a project available called Swarm. It makes use of Scala’s support for serializable delimited continuations. This means you can pause the state of your current program, send it to a different node in your cluster and continue with the program on that machine. And Swarm makes this task extremely simple. So for now a new research project is to take advantage of this feature. We can basically have one master server like Hudson or TeamCity and write a plugin for it that makes use of Scala Swarm. Swarm could be based on nodes in the office, normal computers and maybe a couple of dedicated servers to compile modules in parallel on multiple nodes. Each node would have to keep the VCS up to date. That way we should be able to compile projects much quicker and have a much better workflow in a large codebase. Besides that if you work with a CI server than you could also have the task of code analysis using FlexPMD for instance done on another machine as well.

I am really looking forward to this project when I have some time to work on it. My goal is to develop a plug-in for TeamCity that we will use internally first. I expect to reduce the time for a full rebuild from about 4min to something like 30sec. This would be a huge benefit. I also hope that we will be able to make that plug-in available to the public.

The Advantage Of ActionScript

Published
by
joa
on Feb 11, 2010
in personal
. 9 Comments

Despite the fact that ActionScript has a couple of major glitches and missconceptions I think there are also a couple of great key features that force you to write better applications due to the language design.

First of all we have to aknowledge that ActionScript has some great features. Those include implicit getters and setters or method closures for instance. Java 7 will have method closures finally after a lot of argy-bargy about their complexity et cetera. ActionScript developers are used to method closures and are buddy-buddy with them since Flash 4 I think.

But what is the real advantage of ActionScript? Here are my thoughts.

  1. Poor Performance

    Yes. We have been educated to optimize the hell out of our code. We have been educated to think about our code and how it performs. We have been educated to be creative with our tools, to tinker around. I know that this is not something to be essentially proud of but it led to a lot of creative and astonishing experiments. If you would tell a Java programmer you are still developing object pools they will laugh at you for a good reason. This is where the poor performance is annoying but something is very important: the potential. Think with the mind of a carpetbagger. Shares on the stock-market that never have a downturn are less interesting because there is less profit potential. So why am I talking about this? It is quite simple: Flash has still a lot of potential for growth. Flash did not pass its zenith already like other virtual machines. This makes me very optimistic for the future. There is still so much potential for the growth in performance and I think that Apparat has shown this as well.

  2. Forced Asynchronism

    ActionScript has no ability to perform blocking operations (despite of AIR). This makes every application responsive. If you load files from an external source like a URL your application will stay responsive because you have to add listeners that react on certain events. I know this is somtimes cumbersome but every good application should make use of asynchronous IO for instance. If people are lazy they will simply spawn a new thread to perform blocking operations which is also okay but less performant. Usually the simple example is a server where you would spawn a new thread for every client. This is the worst you can do and Java tackles this issue with its NIO. However a lot of people do not make use of NIO because they think it is hard to use for common tasks.

  3. Events

    Events are a great way to achieve concurrency. Some people call this immutable message passing. If your events do not contain state or are immutable, you have a powerful system to tackle concurrency. Basically this is what the EventDispatcher is all about. You add listeners for certain messages to a dispatcher and then you react on a message when it is dispatched to you. Again we have been educated to use a system which forces us to write responsive applications that is not based on synchronization and locks like Java or C++. If Adobe decides to implement multithreading in the Flash Player I hope they will choose to implement the approach of Erlang which is all about immutable events and event dispatchers. The only difference is that Erlang programmers call them actors.

This post is not an ode to ActionScript. There are still major issues like the lack of parameterized types. But I do think that there is a lot of potential. We can grow with the language and we can tackle upcomming issues. Concurrency is one of them, limited resources on a mobile device are another one.

Infinite Lies

Published
by
joa
on Jan 31, 2010
in personal
. 17 Comments

What you have told us is rubbish. The “open web” is really an acronym for Apple’s cash cow supported on the back of a giant tortoise lie. But Steve Jobs said that Flash is responsible for most of the OS X crashes! You are very clever, young man, very clever. But it is lies all the way down.

The Scalable Apparat

Published
by
joa
on Dec 3, 2009
in personal and projects
. 6 Comments

I am a big fan of Scala and used it for a couple of internal tools here at Hobnox. What I really like about Scala is the fun factor and the programming speed. That is the reason why I decided to port the whole Apparat framework from Java to Scala.

This was not an easy decision since Apparat is a big project. However I have decided to rewrite Apparat because of the following reasons.

  1. Some parts of the framework are not well thought.
  2. TAAS would have been hard to optimize.
  3. This is my (n+1)-th iteration of implementing the ABC format. I think a lot of things can be simplified again.
  4. Scala is an ideal language for Apparat’s tasks.

The Scala source code is already available on Google Code.

Commits will happen early and often. The current revision is able to parse Swf and Swc files. I have also implemented most of the SWF tags which have been part of the Java framework.

I could compare the Scala and Java source every day. Everything is so concise now.

Polyglott Programming On The AVM2

Published
by
joa
on Nov 16, 2009
in Flash, experiments, personal and projects
. 8 Comments

Take some time and think about this tweet for a moment. It took me a while to realize that Joel Hooks is right. I was embaressed of myself. How could I forget about that? But I had also a big smile on my face at the same same time. Let me explain why.

The Java to SWF compiler does not compile Java sourcecode but JVM bytecode to ActionScript bytecode. This means I do not have to teach my program the Java language. It only understands JVM bytecode. This seems like an akward decision on the one hand since working on the bytecode level implies lots of problems. But it turns out that this was a really cool decision on the other hand. “Java to SWF compiler” is maybe the wrong description. “any language that compiles to JVM bytecode to SWF compiler” is maybe better.

So what does any language mean? Here is a list of JVM languages. Now you feel maybe like I did after reading that tweet. And I am really looking forward to get Scala up and running.

Some problems still exist. Threading is one issue and I will basically have to do what Scott Peterson did for Alchemy. But reflections, annotations and method overloading have to be solved as well. Some glitches may exist even after figuring everything out. Stacktraces will look pretty weird. However I think this is a really cool project.

Update: I forgot to mention something important. Java supports native code. This means you can build a library that works with OpenGL for instance. Those native methods can not be converted. There are also some other things that do not work. File access is just one of them.

Getting Rid Of null

Published
by
joa
on Nov 6, 2009
in Flash and experiments
. 16 Comments

A couple of weeks ago I started learning Scala. I can highly recommend it. The language has a lot of great approaches to multithreading and scalability. The reason why I like Scala is because it is so simple yet powerful.

Of course doing something at home influences work. I decided to write a build tool for the Hobnox AudioTool which is about 200 lines of Scala code. Cool thing is that this tool replaces a manually maintained Ant build and all project dependencies are always correct. Plus analyzing the dependencies in a more powerful way allows me to spawn the compiler in parallel. Building the AudioTool is now twice as fast and much more comfortable.

When learning a new programming language you also learn about new concepts. Functional languages in general have a different approach to nullable types. I know Scala is not the only one but let me introduce the concept in terms of ActionScript.

When you have a method that returns either a result or nothing: what do you do? Imagine you have some kind of service and a Dictionary of users. Requesting a user works by his unique id. The Dictionary is private to the class since you want to keep it read only.

function getUser(id: String): User {
  return hasUser(id) ? users[id] : null;
}

If I would now simply ask the service for an unknown user and do something like if(getUser('xyz').isLoggedIn) { trace('Hooray'); } I could probably and up with a null-reference error. No one checks for me if the user exists. So what else could we do? Write a lot of boilerplate code and prepend a check if the user is null or not each time we request one from the service. A much better approach in my opinion is to throw an error as early as possible. In this case we would rewrite the method to something like this:

function getUser(id: String): User {
  if(hasUser(id)) return users[id];
  throw new NoSuchUserError(id);
}

In this scenario we get informed about the error as early as possible. But we are stuck again. First of all ActionScript does not enforce you to catch possible exceptions. This means if you do not read the documentation of a method carefully or look into the source code of a method before calling it somewhere you will never know that it throws an exception. And what if we are actually in a scenario where we do not expect errors for non-existing objects? Think of the Dictionary object throwing each time an error when you access it and the result is null. How could I even check if an object exists in a Dictionary?

try {
  dictionary[key];
  return true;
}
catch(noSuchElementError: NoSuchElementError) {
  return false;
}

I guess you see that this can not be the solution to our problem. In a real world example you may deal with your own collection of objects instead of a Dictionary of course. So we have to get rid of exceptions and null for the optimal solution. Scala’s approach to this problem is the Option type. We always abuse null as a placeholder when we want to express that an element does not exist. The Option means that either Some or None result exists. Rewriting our getUser function using this approach would yield the following ActionScript code.

function getUser(id: String): Option {
  return hasUser(id) ? new Some(user[id]) : new None();
}

Why is this much better than the old approach? When calling the method you will always know that the method has only an optional result value. We get rid of the exception and null values. Our only problem at the moment is ActionScript. The result is now untyped. In an ideal world this method would be written as:

function getUser(id: String): Option.<User> {
  return hasUser(id) ? new Some.<User>(user[id]) : new None.<User>();
}

However we can still tackle this issue by implementing null-representations of our objects. Imagine the User class. You could rewrite the code to something like this.

function getUser(id: String): IUser {
  return hasUser(id) ? user[id] : new NullUser();
}

final class NullUser implements IUser {
  public function get isLoggedIn(): Boolean { return false; }
  public function get name(): String { return 'null'; }
}

And even if you are interested in null-reference errors you could rewrite your code to something like this:

final class NullUser implements IUser {
  public function get isLoggedIn(): Boolean {
    CONFIG::ThrowNullReferenceErrors { throw new NullReferenceError(); }
    return false;
  }
  public function get name(): String {
    CONFIG:: ThrowNullReferenceErrors { throw new NullReferenceError(); }
    return 'null';
  }
}

It is definitely a very different approach. A functional language like Scala allows you to deal much better with Options. But it makes sense to diferentiate between an uninitialized variable which is null and an optional result of a function. Unfortunately this is at the moment very painful with the lack of generics in ActionScript.

TAAS As A Decompiler

Published
by
joa
on Sep 29, 2009
in Flash and news
. 39 Comments

The TAAS compiler is different from the ActionScript compiler since its input is not ActionScript source code but already compiled SWF or SWC files. Just like the haXe compiler can output AS3 instead of a SWF the TAAS compiler can do the same.

Now if you add one and one together you see that the TAAS compiler can be used as a very strong decompiler. My own tests have shown that it will work flawlessly where other commercial decompilers output rubbish. Since the compiler behaves like the Flash Player it will “execute” the bytecode in order to parse it which means it has a very highlevel understanding of the structure inside the SWF.

The only question is now what to do with the source code. I wrote the decompiler for my session at FOTB to show much easier how the optimizations behave. It is also a great tool to debug errors. But should it be opened or not?

To take it one step further one might also be able to write an obfuscator using the TAAS compiler. In my opinion it would be cool to have a strong decompiler and obfuscator, both being open source. We might also add an option to protect SWFs from the decompiler by adding something to the SWF metadata for instance. Of course this is just a simple rule which could be removed by someone once the code is open. What do you think?

FOTB Recordings

Published
by
joa
on Sep 28, 2009
in Flash, news, news and personal
. 14 Comments

The people from Powerflasher were kind enough to upload their recording of my FOTB session. I am sorry that the microphone stopped working at the end. But you might still be able to understand what I was talking about if you turn up the volume of your speakers.
Continue reading ‘FOTB Recordings’

Compiling Java and C# to SWF

Published
by
joa
on Sep 28, 2009
in Flash and experiments
. 13 Comments

At the end of my Leaving the Sandbox session I showed two projects I have been working on without telling anyone. The first was a compiler that could compile C# to SWF. The second was another compiling being able to compile Java to SWF. First of all I have to say that both tools are far from being finished. They are a so called proof-of-concept. For me it is enough to know that this is possible. I am also not sure if I will finish one of those tools.

So how does this work? Because I am now working on the AudioTool backend for our highly anticipated launch I had to switch to fulltime Java development. In order to understand the JVM better I started to search for the optimizations that the HotSpot VM performs and stumbled upon a framework called Soot. Voilà. Reading about Soot made me question why nobody is working on such a framework for ActionScript. Probably because universities see Flash just as a toy. For us developers being dependent on the Flash Platform it is of course more than that.

I will quickly explain what Soot does. It reads compiled Java class files and converts the machine code they contain into a three address code representaiton. This code is then heavily optimized. TAAS is basically doing the same. I also want to note that Soot is by far more complex and complete than TAAS right now. It would be megalomania saying that TAAS is Soot for ActionScript. But you get the idea. The one framework converts Java to three address code and the other framework converts ActionScript to three address code. Maybe it is possible to connect them.

This is what I did for the Java compiler. The whole pipeline looks like this:

  1. Compile Java sourcecode with the Java compiler of your choise
  2. Use Soot to get the three address code representation of a compiled binary
  3. Create TAAS expressions for the expressions that Soot uses
  4. Connect the TAAS graph the same way the Soot graph is connected
  5. Compile the TAAS graph to a SWF

This is about it. There is some glue-code involved here and there and I did not implement all Java expressions yet. I did also not bother about threads. Basically one could do what Scott has done for Alchemy. The cool thing is that we get highly optimized Java code because of the Java compiler and Soot. Then the TAAS compiler can run a second time after linking is done and perform platform specific optimizations like inlining and loop invariant code motion for certain expressions.

I already wrote a couple of classes and packed them in a SWC. Those mimic the behaviour of the Java classes like java.lang.System or java.util.LinkedList. On the Java side I implemented the Flash classes like flash.display.Sprite and flash.events.EventDispatcher. The interesting part is that all the methods those classes contain are marked native which means they have no implementation since they are native to the Flash Player.
This is also a great advantage. It is slow in Alchemy to call Flash methods and to communicate with Flash classes. The Java approach does not have this tradeoff. I also do not have to trigger the ActionScript compiler. The conversion form class to swf is entirely done using Soot and TAAS.

So far for the Java compiler. But in order to show a little bit more of the potential I wanted to have a C# compiler working as well. The great thing about advanced languages and a large community is that people have already built a lot of tools. It did not take too long to find an advanced framework that converts C# to Java. Actually this is not be the best approach. But I could not find the “Soot for C#” the evening before my FOTB session.

So I am using net2java to convert .NET code like C# or VisualBasic to Java. That code is then compiled using the Java compiler and I am using Soot again to convert the code to TAAS without much stress.

For me the most important part is to know that it is possible to compile .NET source code and Java binaries to SWF. To complete those tools one just has to implement all the missing expressions and the standard Java library in Flash. Everything could also work the other way around. Compiling from TAAS to .NET, Silverlight and Java is another option.

After The Sandbox

Published
by
joa
on Sep 28, 2009
in Flash, news, news and personal
. 8 Comments

In 2006 I got invited to the FlashForum Conference in Düsseldorf, Germany. It was my very first Flash conference and my topic was AS3 code optimization. I have talked a lot about bitwise operators, lookup tables and other techniques to speed your code up.
Now about three years later I wanted to prepare a very special talk for FOTB and make a step forward by getting rid of old habits and idioms.

Before FOTB I was not sure if I can present Apparat at all. For me as an autodidact it is not always easy to solve certain issues. Mario Klingemann talked about Magic Moments in Brighton. I guess I had one after a long time of thinking and failing. I got really into the flow and everything became so clear. Stay tuned for more detailed posts on the technique behind the different compilers.

Continue reading ‘After The Sandbox’

« Older Entries