If you ask me if ActionScript is ready for the enterprise then this question is hard to answer. Every fairly huge codebase requires some maintenance. You will have to figure out a strategy to identify modules and separate them. You will also have to handle the dependencies between modules. Last but not least you have to compile those.
An important aspect is that when I am talking about modules I am not talking about “Flex Modules” but the more generic term for a part of a project.
First of all you have to make sure to fulfill some requirements. You can not assume that every developer is working on the same platform using the same IDE with the same settings. But you must assume that your project will not be built from an IDE. You must also assure that the dependencies between projects are up to date across different platforms and IDEs. Once you have figured out how to do that you might want to think about a strategy to compile your modules.
Today I want to explore the idea of parallel compilation across a cluster of machines to yield optimal compile speed for ActionScript projects. First of all, you can not assume that a developer can always keep all modules up to date. Depending on the dependency graph of your projects compilation migh take very long. For instance we have 103 modules at Hobnox. This means we have 103 different compile targets with different dependencies, each yielding its own SWF/SWC file. We also have a “standard library” that is included in nearly every project. Changing this library will require the developer to recompile all 103 modules to update them.
To illustrate this example I want to explain what we went through.
- We used FDT to develop our project. Since FDT is Eclipse based every module was represented as a separate project with dependencies to other projects. FDT however compiles your project only once you launch it. And it does not do this automatically for your dependency graph. As a result you have to manually compile 103 modules in the correct order. This would not be the problem if FDT’s live error highlightning and code analysis would not work on a SWC basis. But since the actual source-code between projects is not analyzed we had to launch all 103 modules to propagate a possible error or change. This was for us the point when we had to make a switch.
- Ant was our second idea to build our FDT projects automatically. However we had to keep module dependencies up to date in all the different Ant files because the FCSH requires that you include external libraries of a library also in a successor of another library. This means if A requires B and C requires A. Then C requires also B. So we had to keep 103 Ant targets up to date for each change.
- A custom build. We started working with a custom build tool I have developed in Scala. This build tool was smart enough to figure out which modules have to be recompiled without launching the ActionScript compiler. It also figured out which projects could be compiled in parallel since the ActionScript compiler is written in a way that IO is not the only bottleneck. We had an enormous improvement in compile time and usability for the developer since module dependencies where handled automatically.
- We switched from FDT to IntelliJ IDEA since it offered us more features that we needed. IntelliJ came also with a much better strategy to compile modules. Basically it had the feature we always missed for FDT to automatically include libraries and to compile modules in the correct order automatically. However the guys at JetBrains made a mistake by counting on FCSH. The short story is that FCSH is absolutely not capable of handling a large codebase. What happened to us was that when we started a build FCSH would take up as much memory as it could, getting much slower until it finally crashed. Then FCSH started from scratch, consumed the maximum amount of memory again and crashed. This happend until a project was finally compiled but took also more than 15 minutes for a single change we made. This was of course unacceptable. Fortunately JetBrains implemented a feature request from us into IntelliJ IDEA which comes close to our custom build. From now on projects are compiled in parallel without using FCSH and without asking launching the compiler if no change has been made. This way we are able to work again. However only if we compile just a short amount of all modules since the main bottleneck is still the compiler.
- A continuous integration server was our last option. This is a custom build server and we are using it until today. After each SVN commit we are building the required modules and their dependencies now automatically on a dedicated server. A change in the standard library takes about four minutes to complete now on a quad core. But the good thing is that we have set our main project to exclude a lot of libraries. The custom build takes care of that and we are loading now modules from the server. That way we can still compile local and include libraries we frequently change. Other libraries are loaded from the server and included at runtime.
So the question is if this is now the ultimate solution. A custom continuous integration server that takes more than 4 minutes to propagate a change after a SVN commit? I do not think so. And here comes Scala into the game again. There is a project available called Swarm. It makes use of Scala’s support for serializable delimited continuations. This means you can pause the state of your current program, send it to a different node in your cluster and continue with the program on that machine. And Swarm makes this task extremely simple. So for now a new research project is to take advantage of this feature. We can basically have one master server like Hudson or TeamCity and write a plugin for it that makes use of Scala Swarm. Swarm could be based on nodes in the office, normal computers and maybe a couple of dedicated servers to compile modules in parallel on multiple nodes. Each node would have to keep the VCS up to date. That way we should be able to compile projects much quicker and have a much better workflow in a large codebase. Besides that if you work with a CI server than you could also have the task of code analysis using FlexPMD for instance done on another machine as well.
I am really looking forward to this project when I have some time to work on it. My goal is to develop a plug-in for TeamCity that we will use internally first. I expect to reduce the time for a full rebuild from about 4min to something like 30sec. This would be a huge benefit. I also hope that we will be able to make that plug-in available to the public.
If you ask me if ActionScript is ready for the enterprise then this question is hard to answer. Every fairly huge codebase requires some maintenance. You will have to figure out a strategy to identify modules and separate them. You will also have to handle the dependencies between modules. Last but not least you have to compile those.
An important aspect is that when I am talking about modules I am not talking about "Flex Modules" but the more generic term for a part of a project.
First of all you have to make sure to fulfill some requirements. You can not assume that every developer is working on the same platform using the same IDE with the same settings. But you must assume that your project will not be built from an IDE. You must also assure that the dependencies between projects are up to date across different platforms and IDEs. Once you have figured out how to do that you might want to think about a strategy to compile your modules.
Today I want to explore the idea of parallel compilation acr
I just want to share with you how Taas can actually optimize code and what happens behind the scenes. So in my last post I talked about stackless code and optimizations that are possible but how does it work at all?
Imagine you have got this ActionScript code:
var x: Number = 1.0;
var y: Number;
if(true)
{
y = 3.0;
}
else
{
y = 2.0;
}
x += y;
return x;
For the sake of simplicity I will not use local variables in the bytecode. But it will be easier to figure out what happens in the bytecode.
PushDouble 1.0
PushTrue
IfTrue L0
PushDouble 2.0
Jump L1
L0: PushDouble 3.0
L1: Add
ReturnValue
This bytecode is nearly the same as the method but without local variables. This should be fairly simple to understand.
When converting bytecode to Taas, its corresponding control flow graph is converted into a control flow graph of Taas expressions. I do not want to go into much detail how the stack based code is converted but here is the graph before and after the transformation from bytecode to Taas. It looks quite similar but there is a major difference. All those constant values in the Taas graph are not values pushed on a stack but assignments to virtual variables which exist only in theory. You can see also that the Add in the bytecode has known operands and type in the Taas version. Since it is currently not know if 2.0 or 3.0 has been used there is a Φ function that says “This value is either 2.0 or 3.0 depending on the path taken at runtime”.
So as I said this code can be optimized much better than stack based code. This graph is very redundant. There are three utilities to compact and simplify the current graph of Taas expressions. The algorithm has been developed for Java bytecode but works with ActionScript very well tool. The concept is quite simple. Perform copy propagation, constant folding and dead code elimination until the graph stops changing. Applying these techniques to the Taas graph yields the following results.
Most of the theoretical variables have been eliminated using copy propagation.
Known constants have been replaced. Even the if condition is no longer needed and the false branch has been removed. The dead code elimination will clean up this code afterwards.
Dead code elimination has removed the dead Jump statement and also the value 2.0 from the Φ expression. Afterwards constant folding replaced the Φ expression with its one constant value. Then another iteration of constant folding replaced the Add expression with the constant value 4.0. Afterwards copy propagation has put the result into the Return statement and voilá.
This result may have seemed quite obvious from the beginning and you may ask who writes such code. Probably nobody. But once you start inlining methods this makes a lot of sense. A lot of preconditions are known in that case and unnecessary branches can be removed. Since inlining methods bloats up the code it is very important to compact it afterwards as much as possible. I hope you see now how interesting all of this might be in the future.
I just want to share with you how Taas can actually optimize code and what happens behind the scenes. So in my last post I talked about stackless code and optimizations that are possible but how does it work at all?
Imagine you have got this ActionScript code:
var x: Number = 1.0;
var y: Number;
if(true)
{
y = 3.0;
}
else
{
y = 2.0;
}
x += y;
return x;
For the sake of simplicity I will not use local variables in the bytecode. But it will be easier to figure out what happens in the bytecode.
PushDouble 1.0
PushTrue
IfTrue L0
PushDouble 2.0
Jump L1
L0: PushDouble 3.0
L1: Add
ReturnValue
This bytecode is nearly the same as the method but without local variables. This should be fairly simple to understand.
When converting bytecode to Taas, its corresponding control flow graph is converted into a control flow graph of Taas expressions. I do
In September 2008 I started working on a tool called AS3V. The goal of AS3V was to parse source code and to check the code against a set of rules. My main problem with the first attempt of AS3V was that I used ANTLR. ANTLR is a great tool to create a compiler but for ActionScript 3 and its various strange behaviours you have to do a lot of hacking. Parsing ActionScript, XML and regular expressions with the optional semicolon in ECMAScript is not really fun using such a tool. metaas went that path and unfortunately their parser fails very often.
This is the reason why I started writing a parser manually from scratch. But this is also not a good idea. What about Adobe changing the language or adding features? I would have to change my parser. This is for instance a problem you have with FDT. Adobe releases a feature and it will take a while until you can adopt that. Maybe that feature is only beta but you want to use it. So are you going to implement it or not? Since the Flex SDK is open source I decided to take a completly different approach. Option one is to modify the original ASC. I think this is a bad idea. On every update from the Adobe guys you would have to patch the compiler again, and release a compiled version of that. This is what flexcover does. But I do not want to replace core parts of the Flex SDK with a modified compiler. So this is the reason why AS3V links only against the ASC. This means basically I do not care about what Adobe does with the compiler as long as the API stays consistent. This means also AS3V knows only as much as the ASC. This is sometimes a problem but in most cases not so important. ASC drops for instance braces every time. AS3V does not know if you write if(true) trace('foo'); or if(true) { trace('foo'); }. But in the end AS3V is able to analyze the real compiler output which is huge; theoretically every implicit cast can be watched!
Now enough theory. Why bother using AS3V and what is it good for? Imagine a loop like this:
for( var y: int = 0; y < height; ++y )
for( var x: int = 0; x < width; ++x )
index = y * width + x;
AS3V is currently smart enough to analyze the statement y * width + x and to figure out that y and width do not change in the inner loop. AS3V is therefore able to warn you about unneccessary calculations. There are lot of other rules I have already implemented which help you writing better code. For instance the pattern if( expr0 ) { if( expr1 ) { foo(); } } can be reduced to if( expr0 && expr1 ) { foo(); }.
There is a lot of room for other rules. I am happy to see some progress and hope that other people will find AS3V as useful as I. In the current state I have to implement more rules and a little bit more complex ones like dead code detection with constant folding. A release is targeted once I am happy with the rule set.
Not to forget: An Eclipse plugin which will show you appropriate warnings as neat little icons is of course also planned.
In September 2008 I started working on a tool called AS3V. The goal of AS3V was to parse source code and to check the code against a set of rules. My main problem with the first attempt of AS3V was that I used ANTLR. ANTLR is a great tool to create a compiler but for ActionScript 3 and its various strange behaviours you have to do a lot of hacking. Parsing ActionScript, XML and regular expressions with the optional semicolon in ECMAScript is not really fun using such a tool. metaas went that path and unfortunately their parser fails very often.
This is the reason why I started writing a parser manually from scratch. But this is also not a good idea. What about Adobe changing the language or adding features? I would have to change my parser. This is for instance a problem you have with FDT. Adobe releases a feature and it will take a while until you can adopt that. Maybe that feature is only beta but you want to use it. So are you going to implement it or not? Since the Flex SDK is open source I decided to
First of all I think I have to clarify at least one thing. I have criticised Adobe in the past for a lof of reasons. Not because I do not like them or the technologies they produce but because I want to improve the Flash Platform. This is of course pure self-interest since Flash is a key technology for the 
