I am a big fan of Scala and used it for a couple of internal tools here at Hobnox. What I really like about Scala is the fun factor and the programming speed. That is the reason why I decided to port the whole Apparat framework from Java to Scala.
This was not an easy decision since Apparat is a big project. However I have decided to rewrite Apparat because of the following reasons.
- Some parts of the framework are not well thought.
- TAAS would have been hard to optimize.
- This is my (n+1)-th iteration of implementing the ABC format. I think a lot of things can be simplified again.
- Scala is an ideal language for Apparat’s tasks.
The Scala source code is already available on Google Code.
Commits will happen early and often. The current revision is able to parse Swf and Swc files. I have also implemented most of the SWF tags which have been part of the Java framework.
I could compare the Scala and Java source every day. Everything is so concise now.
Take some time and think about this tweet for a moment. It took me a while to realize that Joel Hooks is right. I was embaressed of myself. How could I forget about that? But I had also a big smile on my face at the same same time. Let me explain why.
The Java to SWF compiler does not compile Java sourcecode but JVM bytecode to ActionScript bytecode. This means I do not have to teach my program the Java language. It only understands JVM bytecode. This seems like an akward decision on the one hand since working on the bytecode level implies lots of problems. But it turns out that this was a really cool decision on the other hand. “Java to SWF compiler” is maybe the wrong description. “any language that compiles to JVM bytecode to SWF compiler” is maybe better.
So what does any language mean? Here is a list of JVM languages. Now you feel maybe like I did after reading that tweet. And I am really looking forward to get Scala up and running.
Some problems still exist. Threading is one issue and I will basically have to do what Scott Peterson did for Alchemy. But reflections, annotations and method overloading have to be solved as well. Some glitches may exist even after figuring everything out. Stacktraces will look pretty weird. However I think this is a really cool project.
Update: I forgot to mention something important. Java supports native code. This means you can build a library that works with OpenGL for instance. Those native methods can not be converted. There are also some other things that do not work. File access is just one of them.
I have developed some other optimizations during the past couple of days. Including strength reduction and optimizing tail recursive calls.
Strength reduction can already handle about 55 different cases at the moment. For instance it will convert code like if( x - 1 == 0 ) into if( x == 1 ). But it will also remove expressions that other optimizations could introduce. x + 0 is something that might occur during inline expansion. Or x + Number.NaN for example.
But I really like the tail recursive optimizations. TAAS can detect if a method needs to call itself or not. Take this code for example:
private function sum( i: int, value: int ): int
{
if( i == 0 )
{
return value;
}
return sum( i - 1, value + i );
}
You see that the last statement sum( i - 1, value + i ) is tail recursive. This means we can replace the recursive call to sum() with a jump to the beginning of the method after changing the method parameters. The sum() method becomes converted to something like a simple while loop. This means also that one could have written code like this in the first place since this optimization applies only to methods that do not have to be recursive after all.
Knock yourself out with some examples:
Finally I am able to present the first results of TAAS. It was a long way to get here. I was actually not sure at all if I will manage to show something at FOTB 09. Let me explain what happens here.
In the image is the original ActionScript code on the left. The ActionScript bytecode produced by the ASC is in the middle and the compiled TAAS version is on the right. You see that I am running a loop and call a method inside that loop. It would be nice to have small helper methods like this one inlined automatically. TAAS can do this for you now.
So how does this work? First of all an SWF file is parsed and the bytecode is extracted. This bytecode is transformed into a control flow graph. This control flow graph is converted into a graph of TAAS expressions which a lot of benifits. The conversion step works like the Flash Player and kind of executes your code. The result is that all methods and local variables are typed if that is possible.
It is much easier to perform a lot of optimizations now. Like inline expansion. In order to do that I can simply compile the method that is called to TAAS and connect the vertices together in the graph after some adjustments have been made. The inline expansion step will automatically inline methods when it thinks it is useful and possible. You can only inline methods that are private or final and make no use of reflections.
Inlined methods create a lot of overhead. Usually they contribute additional local variables and the code gets bigger. So one step after the inline expansion is to clean everything up again. In this example I can remove those registers thanks to copy propagation and dead code elimination.
And it gets even better. If you take a closer look at the example, the calc method expects two paramters that are of type Number. Therefore the original method uses an Add instruction. TAAS knows about the types and sees that it will pass two integers into this method. Since it makes no sense to convert from int to Number to int it stays with integer in this case.
So after inlining the method ends up with even less local variables. The original local variable t1 is considered useless since it is only used one time. TAAS will put the code getTimer() at the position where t1 has been used and we end up with a heavily optimized method. What I like the most is that I do not have to change the way I write my ActionScript code. All happens behind the scenes automatically.
Some other optimizations are implicit. TAAS will use AddInt instead of Add if both operands are typed int. A FlowOptimizer is also involved and will invert the if expression for the loop. This can reduce the number of required jumps in a method by 50%.
I know that the output is not perfect. But this is all a work in progress and really the first result that I can share. The SWFs speak for them self.
Dump has now another cool feature: UML diagram generation. The UML diagram is exported in DOT format. I think this is a really cool feature because the graph is built by analyzing a SWF file and you get it for free. I will probably create a different tool to make full use of the UML generation since you could link multiple files together for a complete coverage. Since Graphviz is not able to underline text I have choosen to use a dollar sign for static methods. You will also get proper parameter names if you compile your SWF file in debug mode.
This is an example UML diagram for the playerglobal.swc. In order to create it I took the playerglobal.abc from the Tamarin sources and the command was java -jar dump.jar -input playerglobal.abc -uml.
Here are two example representations but be careful. The PNG size is 30831×6232 and might crash your browser. Chrome can display the PNG for me but is not able to show the SVG correct. Firefox displays the SVG very well. You can download the PNG and open it in IrfanView or Photoshop to be safe.
The Dump tool is now able to export an inheritance graph for a given ABC/SWC/SWF file. This is a very easy and nice way to look at the classes and their relationships. The small image shows the graph for one ABC file of the AudioTool. I think this it is pretty neat.
You will need a program like Graphviz to visualize the exported DOT file. If you want to export the inheritance graph you basically writejava -jar dump.jar -input file.swf -ig. I think this shows a really cool feature of Apparat. It is also very easy to reverse engineer a UML diagram. I am not interested in such a feature but maybe someone else.
Update: Here is a full-size example for the inheritance graph of the unfinished ImageProcessing library I am currently working on.
A very important tool for SWF manipulation is a way to debug them and to have a look at the bytecode. Dump is a tool that does nothing else but listing all SWF Tags and ABC files the way Apparat represents them internally.
I know that there are a lot of other tools out there. swfdump and abcdump are great already. I have used swfdump from the Flex SDK before. But I dislike how they represent namespaces and the fact that not all properties are always shown. Dump is simple. It lists everything and uses the naming form the avm2overview.pdf.
The full source code of Apparat is now available at GoogleCode. It is the whole framework behind TDSI and Reducer.
Apparat is released under the GNU Lesser General Public License. Please contact me if you want to contribute to this project. Maybe someone is interested in writing an Ant task for Reducer? I am also happy to receive feedback if you have used the framework to build something cool with it.
Please join the Apparat Discussion Group if you are interested in collaboration.
Another spinoff from my current library to optimize SWF/SWC files. Reducer is a tool that will make SWF and SWC files significantly smaller. There is currently a huge problem with filesizes. If you use the [Embed] tag with PNG images they are not compressed at all. For the Hobnox AudioTool we have been using the Flash IDE to export all graphics so that they are smaller which was a pain.
Now with Reducer you are safe to use [Embed] and then run the tool afterwards. It will compress all lossless images and make them lossy. But usually a PNG can be compressed at 100% JPEG quality and you will still safe a lot of data.
Note: You will not loose alpha transparency when using Reducer. The SWF file format allows us to use a special compression where a PNG gets split up into its color and alpha channels. The color channels are encoded using the traditional JPEG algorithm with adjustable quality while the alpha channel is handled seperately. Transparency is always stored in a lossless fashion which means even with a low JPEG quality you will not get any compression artifacts for the alpha channel.
Update: Reducer is now open source!
I am definitly not good at choosing names for software projects. However TurboDieselSportInjection is a release of my experiments from yesterday. It is a spinoff from the whole framework and allows you to inline __bytecode and of course to use the new Memory API.
Hopefully you are kind enough to provide me with some feedback. I am especially interested in Exceptions that occur when reading or writing ABC files. Have fun!
Update: TDSI is now open source!
