Wikipedia defines STM as:

In computer science, software transactional memory (STM) is a concurrency control mechanism analogous to database transactions for controlling access to shared memory in concurrent computing.

First of all when reading this you might think that an STM is not necessary since ActionScript does not support true concurrency and also the shared-nothing worker proposal will not allow you concurrent memory access. While this holds true you may not forget that concurrency becomes an issue when you have just access to a single core but on multiple machines.

A lesser known fact about Audiotool is also that we have a built-in system for real time collaboration since its beginning. Due to conceptional issues and other prioritized features we were never able to roll it out as a full feature but the technology base is there.

When it comes to multiplayer gaming and network latency you will always have to figure out a system which allows you to share a certain state across multiple machines. In some games the state is synchronized across a certain server. An example is an FPS like Quake3. The Audiotool is unlike Quake3 not a FPS and different conditions have to be met. And this is the reason why I have choosen to implement an STM with very optimistic concurrency control.

The basic idea of my system is to use atomic transactions with commit and revert operations. Those are handled by a transaction manager which performs the basic tasks of an STM. That means: execute all transactions until an error occurs, roll back and try again. Each transaction in the Audiotool contains a guard. The guard defines whether or not one should still execute the transaction.

Here is an example of why a guard is important and how all this stuff works. You have two Audiotool instances running that share the same state. We call them A and B. A and B are connected via P2P no an arbitrary network. This means exchanging messages introduces latency and we cannot guarantee that A is at the same state of B.

So what happens? t₀ is committed by A, serialized and sent to B who is committing the same transaction. The same applies to t₁. t₂ is initiated by B, serialized and sent to A. Everything works fine until B commits t₄. While t₄ is being transferred to A, A created t₅ and committed it. This means A never saw t₄ when it committed t₅ and B would continue with t₆ ignoring t₅. A will actually receive t₄ after he commited t₅ so the system will revert t₅, commit t₄, commit t₅ if the guard does not prevent it, and both continue at t₆.

To explain the guard: Imagine the t₅ transaction includes modifications on a device in the Audiotool that is deleted by t₄. In that case t₅ will never be executed.

So what does this imply? You can guess that each modification of the Audiotool state is in fact handled by a transaction and we carefully have to pick the guards. E.g. if an editor is referenced by a transaction it is part of the guard. However this introduces a second issue: references. Sharing references of objects across a network is a tricky task. I am using “boxes” to solve this issue. A box can hold a value and it is guaranteed that it will hold the same value at the same state across the network. A box can be serialized and will be shared between commit and revert states. This means if t_n creates an object which is referenced by t_n+1 both will use the same box. This also means that I can safely revert t_n+1 and then t_n and commit them without having to worry that both still reference the same object. And this even works across the network. And in fact we can now also identify whether or not one transaction would conflict with a different one which is a very important condition for the actual implementation.

Right now you might have figured out that a transaction in the Audiotool has the following properties:

• It is serializable so it can be transferred across the network.
• All object references must be stored via “boxes”.
• A guard is used to determine whether or not to commit a transaction.
• Transactions can be committed and reverted.

A lot of stuff one has to reason about. But although we do not support live collaboration at the moment we are using the STM. Why? The reason is quite simple. The STM gives us history for free. Since everything is serializable via a very lightweight protocol we can take the last 500 transactions for instance and store them as a compressed ByteArray in memory instead of each as a single object with all its references.

Another very nice option is testing. A serializable history means that one simply has to press a super secret shortcut to dump the whole history and send it to the responsible developer who can reason about every single step that happened which ultimately led to the error. In development mode I usually commit, revert and commit each transaction. Then I also serialize and deserialize them and do the whole thing again.

Since transactions are known by the system we can also write a fuzzer that executes lots of crazy stuff. Sort of Android’s monkey mode. And ultimately I personally hope that we will get to the point where we will enable live collaboration in the Audiotool of course.

The last feature I implemented is the automatic transformation of arbitrary transactions to a compound transaction. You can guess what this feature actually means for the user :). In fact it is very complex and I implemented it with the STM using only a couple lines of code (of course that is just half the story …).

If you are interested in writing your own multi user application I think this is a very nice approach since collisions do not happen very often from my experience and the optimistic concurrency in a P2P environment achieves great results in terms of overall latency and feeling. The Google Wave protocol and their operational transform is also an interesting read.

My talk at FITC San Francisco is over and I want to share some of the anouncements from today with you. At the end of my talk I was showing JITB.

What you see in the YouTube video I posted a while ago is a Java program executing a SWF. For FITC I added some more code and an OpenGL based Display List renderer. In other words: I wrote a Flash Player.

However I should rephrase that statement and say I am attempting to build a Flash Player. The current state is available in the sf2010-sprint clone of Apparat. I will merge the changes into the main Apparat branch when I am back home in Germany.

JITB is currently able to translate a subset of ActionScript code at runtime into Java bytecode and runs nearly at the same speed as native Java. This is a really huge improvement compared to standard ActionScript performance. A lot of smart people worked on the JVM and made it really fast. Apparat will allow you to leverage all this hard work in the future. I am also shooting for Java interoperability at some level so that you can call Java classes from within ActionScript. Hopefully you will be able to use JITB on your desktop machine, on a server or on an Android phone. Basically everywhere Java runs.

There are still a lot of things missing. The whole Flash API needs to be implemented. And the Display List rendering needs a proper OpenGL implementation. However I thought this might be some cool stuff to share with you in its early stages.
My hope is that more people start contributing to the project. Maybe some OpenGL guru wants to take care of the Display List rendering or someone else likes to help implement the Flash API in Java.

I also showed a Raytracer by Nico Zimmermann during my presentation and promised to put the URL on my blog so here it is. His company is called Britzpetermann and the address is http://www.britzpetermann.com/.

Update: Please do not think that this implementation is 30x faster than the Flash Player developed by Adobe. One(!) microbenchmark is never a number you should count on. I would like to make clear that I never said this.

I know it would be better to have it as an Eclipse Plug-in with nice little markers etc. but for that I need some more extension points in FDT. I do not know about FlexBuilder but it would be probably the same.

The file includes an example build and some test cases that show you how AS3V can detect common errors and mistakes. There is currently not that much documentation available for the rules and parameters you can specify. Putting a list online with all rules and nice descriptions is of course something I would like to do but currently the time for that is missing. I hope everything is self-explaining if you have a look at the readme and example files.

• Download AS3V

Update: The AS3V package has been updated with one bug fix and some modifications so that AS3V will produce a more interesting log output if it fails for some reason. Using AS3V with Eclipse works only if you specify fork="true" like in the Flex SDK example from the build file that comes with the zip.

See it as a knowledge base for ActionScript optimization techniques, data structures and code snippets. I would appreciate any feedback and suggestions for articles in those or other categories.

• ActionScript Wiki

You will always have problems without some kind of raster. Sometimes more, sometimes less. Usually you have to build your own context of how things are connected and how they behave in your environment. We were discussing a lot about the audio engine and the core structure behind. The most important part of the audio engine is probably the ability to build a linear chain of plugins which process signals that arrive in your speakers at the end of that chain. When we started this part we had a lot of optimizations in mind. Plugins that do not generate any audible output should not go into that chain etc. The code became quite complex and results in a lot of special case scenarios. Imagine a scope which is a pure visual element that shows the incomming waveform. This is a plugin which should not be kicked out of any chain but generates no audible output.

You can imagine that we used a graph for that kind of task. To be more specific: a directed cyclic graph since there is a direction of the flow and we are allowing feedbacks for audio signals. After some time of discussion I started researching for a better way of solving our signal chain. Because I beleive that a problem can be solved the easiest way on a pure mathematical basis I started with a plain graph — but this time a real graph. After some time I could produce the same output as our old solver but with less lines of code and we got rid of all the special cases.

Now having a real graph is fun and I was not surprised that lots of people invested their brainpower in graph algorithms. For the desktop in the AudioTool I always wanted to implement some kind of automatic layout algorithm. My first idea was to build a spring between two editors connected by a cable. I thought that this could be too complex but after spending the last days with graphs this became a possible solution and there are already algorithms which do the same thing: build springs for connections and let the system solve itself. I did some tests in a sandbox already but hopefully I can implement this in the AudioTool before FITC Amsterdam. The graph you see in the picture is a result of the graph layout engine.

But working with graphs opened my eyes even more. I am currently the poor bugger that has to implement a layout engine. Implementing a layout engine is by the way the most ungrateful task I could think of. Anyways, what if you think about the dependencies in your layout engine as a graph structure? You have usually three cases when calculating a size: relative known size to “some kind of available size”, absolute known size, absolute unknown size. Those cases can be reduced to their most simple form on a piece of paper and a graph structure evolves. You can get rid of observers and listeners in such a system if a connection in the graph has the meaning of invoking the layout engine again on a subset of nodes. Those subsets could be optimized even more if you figure out the strongly connected components in the graph. I have to admit that I did not implement this system yet but it makes a lot of sense in theory currently… something which makes working on a layout engine interesting again ;)

But there is still a big problem when it comes to dealing with image data in Flash. I have currently different strategies with their pro’s and con’s but I can not decide for myself which one makes the most sense.

1. BitmapData

   When you are using only one BitmapData to represent an image you have of course access to all the native methods. Life will be a lot easier. But there are certain key issues with the BitmapData. The precision for each channel is limited to 8bit and the main problem is that you will always have to deal with pre-multiplied alpha. If I want to have a high-quality image processing library this is unacceptable.

2. Vector.<uint>

   A vector filled with unsigned integers makes sense because this is what you get from a BitmapData and what you can use to set the contents of a BitmapData but working all the times with one integer value and having to do the bit-hassle is really annoying.
   The main problem with a Vector of unsigned integers is simply that PixelBender does only accept a Vector of Number values with the length of width * height * 4. So everytime you want to use PixelBender you would have to convert your vector. And also remember that you have only a precision of 8bit per channel. Instead of using a Vector.<uint> I could also use a BitmapData — the only thing I gain from the vector is that I do not have to deal with pre-multiplied alpha.

3. Vector.<Number>

   A vector of normalized Number values makes a lot of sense. You are able to use PixelBender with this kind of Vector without having to convert anything. The only problem occurrs when you want to convert your vector into a BitmapData — but this problem can be solved pretty easy with a PixelBender identity shader. This is also what I used to convert a BitmapData into a vector of Number values since BitmapData.getVector returns only a vector of unsigned integers.

   Now all of this does not sound that bad. But there are a lot of problems actually. First of all every time you want to get/set a pixel you have to access four fields of a vector which is way slower than accessing only one field. And the main problem I have is the poor PixelBender support. First of all every time I start a ShaderJob with a vector of fixed length as the target I get an exception that the ShaderJob.start() wants to change the length of my vector. What the fuck? If my Vector has a fixed length of width * height * 4 I do not see why the ShaderJob should change this at all.

   Another problem is PixelBender itself. In order to rebuild a method like BitmapData.fillRect() you have to visit every channel for every pixel if the rectangle is the full image size. Using ActionScript for this task is out of the question because it will be way to slow. So I thought I write a fillRect-Shader. No problem. The shader works very well in the PixelBender Toolkit but I get only garbage in the Flash Player. Now I am really uncertain if I should continue with this approach since even simple tasks fail. If my target is a BitmapData it works very well by the way. So if I want to fill a rectangle with PixelBender and I have only a Vector.<Number> I have to create a BitmapData with equal size to use it as the target with the ShaderJob and then use an identity shader to convert the BitmapData back into a Vector.<Number>. And once we are using a BitmapData we have again lost the precision and we have pre-multiplied alpha. This would not be a problem if PixelBender would work with Vector.<Number> as good as with a BitmapData.

4. Vector.<RGBA> and single-linked list

   Having a Vector.<RGBA> in combination with a single linked list is by far the best what you can get in terms of speed. We use this in our audio engine as well. You need to access every field only once and you can iterate over the single-linked list very fast. There are only two major issues. In order to use PixelBender you have to convert this structure which has the length width * height into a Vector.<Number> of the length width * height * 4 and you will have the same problems as described above. If you want to have a BitmapData representation you will have to convert the Vector manually to a BitmapData which will also cost you a lot.

My main problem is that I do not like any of those possible approaches. In the old ImageProcessing library I used a BitmapData for each channel but I would not do that again. The main problem is that Adobe has released a very buggy version of the PixelBender run-time in Flash. I would criticize also that we have so many different formats for pixel data which do not fit together. Sometimes you need a Vector.<uint>, sometimes you need a Vector.<Number> — sometimes the endian is flipped (Adobe Alchemy) and working with the BitmapData only — which sounds reasonable — is a pain because you will never get around pre-multiplied alpha. I would love to have a primitive RGBA data type for image processing which I could feed into PixelBender (color4) and which I could get and set to a BitmapData as a Vector.<RGBA>. If those RGBA elements would be also a single-linked list — even better.

I want to a step further now. But please keep in mind that I am definitly not a compiler expert or have deep knowledge of the Flash Player. Everything I write about is in my opinion common sense and comes from a long time of reverse engineering and autodidactic learning.

Alchemy

First of all I have to say that Alchemy is a great research project and I have full respect for Scott Peterson and his work. He did an awesome job and I am more than happy that Alchemy has now been released on labs because people begin to question now the Flash Player, ActionScript and the ASC (ActionScript Compiler).
So why is Alchemy faster than using plain ActionScript — or is that true at all? I do not want to disappoint anyone here but writing C/C++ code instead of ActionScript is not the right way. Alchemy will never be faster than your plain handoptimized ActionScript and it still has to prove for me that it is actually faster. All my comparisons between Alchemy and ActionScript showed that using ActionScript is in most cases faster.
But it makes sense to use Alchemy. An audio encoder for instance runs for a long time on its own without having to exchange data. Currently it is very slow to transfer data between Alchemy and ActionScript on every frame. This does not mean Alchemy is bad, but it will not work for your fast inverse square root algorithm.
So why can Alchemy be faster at all? There are a couple of reasons for that.

1. C/C++ can be compiled and optimized much better than ActionScript
2. The language offers you features you would dream about as an ActionScript programmer (operator overloading, method inlining, templates, etc.)
3. The LLVM does a really good job on optimizing its intermediate language
4. Alchemy comes with some new opcodes for the Flash Player that allow very fast ByteArray operations
5. Alchemy uses a modified ASC that supports inlining of Flash Player bytecodes.

So just to make one thing cristal clear: If the new opcodes would not exist, Alchemy would produce very poor results in terms of execution speed. But albeit that I have to disappoint you about the execution speed of Alchemy in general. Nicolas Cannasse wrote an interesting blog post about this issue and I completely agree. ActionScript does not support a lot of features which C/C++ offers — pointers for instance. All those features have to be wrapped and this means a significant speed loss.
But I do not want to start an Alchemy bashing here. I still think it is a really great technology and the main use case is not fast code execution. But it brings me to my next points: The ActionScript language and the ASC.

ActionScript

ActionScript has been designed several years ago and Adobe had always the incentive to be ECMA compilant. Being standards compilant reduces a certain level of innovation and recuded the power of the Flash Player in the last years. Right now, ActionScript 3 is not an ECMA compilant language. And I am not saying this because ECMA Script 4 has been dropped. ActionScript 3 added additional “features” even before that happened.
For a long time ActionScript 3 has been very close in terms of its ECMA compilance but I think it is now time for innovation. The ActionScript 3 language does not fulfill most of the professional needs.
Have you ever looked at haXe? I hope you did. haXe is definitly not perfect but I want give you a small excerpt of haXe language features which will make your mouth water:

• You are able to use type parameters
• You are able to use function types like Event -> Void for an event listener instead of Function
• Enums instead of the public static const convulsion

Why am I talking about haXe at all? Java has most of those features implemented as well and several other languages too. But haXe can output a SWF so we can compare it to ActionScript. haXe is in a lot of ways superior to ActionScript and I did not talk about the compiler yet.
The quintessence is that ActionScript is not a feature rich language compared to mature ones. And even a single French genius can develop something better than a multi-million dollar company. What the hell is going on here?
The main problem of ActionScript is that you lose most of your types along the way and therefore the compiler can not produce a good output. E.g. everytime you put some object into an Array in ActionScript its type is lost unless you cast of course when reading again — and I hope you do that. This is a really big problem. This is the reason why Alchemy can produce “better ActionScript” than you. Not because you are stupid but because you do not have the tools in ActionScript to do that. And this is a shame for Adobe. So many great developers could produce such a great output but they are not allowed to.

ActionScript Compiler

One of the main problem in the whole Flash universe is the ActionScript compiler. This is a little bit harder to explain because it is a little bit more abstract. Some problems exist because of the ActionScript language nature. But some problems exist because of the compiler itself. Let me explain this with an example:

package {
public final class Example extends flash.display.Sprite {
public function Example() {
var a: int = 1+2;
var b: int = a;
var c: int = b+1;
}
}
}

This code will produce exactly this output for the constructor:

get_local0
pushscope
get_local0
constructsuper 0
pushbyte 1
pushbyte 2
add
convert_i
set_local1
get_local1
convert_i
set_local2
get_local2
pushbyte 1
add
convert_i
set_local3
returnvoid

The interesting part has been marked bold. Let me explain what is going on here:

1. Put value 1 on the stack
2. Put value 2 on the stack
3. Add the two values which are on the stack together using floating point calculus
4. Convert this floating point value to an integer
5. Set this value to local register one which is a
6. Put the value of local register one on the stack
7. Convert this floating point value to an integer
8. Set the value to local register two which is b
9. Put the value of local register two on the stack
10. Put value 1 on the stack
11. Add the two values which are on the stack together using floating point calculus
12. Convert this floating point value to an integer
13. Set this value to local register three which is c

Basically this is an example for the worst output a compiler could produce. First of all b is not needed at all in our code. We hold just the value a in b. a holds a constant value 3. Why do we have to add this stuff together at runtime everytime if it is constant? For c we add the constant value 1 to the constant value 3 which has been stored in a. The compiler should know at this point three things: a is constant, b is not needed, c is constant. The compiler could just set c to the integer value of 4. This is also what I would expect for a SmallWebFormat.
But to make it even more interesting: c is not used at all. So why does the constructor contain any bytecode at all besides calling the super constructor? And to make it a little bit more absurd: Try doing the same in PixelBender and use Tinic’s disassembler. You will see that you will not find any of your local variables in the PixelBender bytecode — which is correct. And the answer why the PixelBender compiler is smarter than the ASC is quite easy: Adobe uses the LLVM for PixelBender as well. Remember the LLVM is what Alchemy is using. This means basically that PixelBender has a better compiler than we have for ActionScript. This is sad and insane but true.
With the haXe compiler you will get nearly the same output as the ASC with certain differences. All the variables are integer values and therefore haXe uses integer calculus. And the last addition does not involve b at all. haXe will not ignore that a, b and c are not used at all. But as I said: haXe is definitly not perfect — but it is by far better.

I have asked for certain features a lot in the past and implemented them for myself. I asked Adobe to support the __asm keyword even before Alchemy and they refused it. Of course I am a little bit disappointed about that because now they use it for Alchemy and they see it makes sense. Basically having a __asm keyword allows you to get around the nonsense which the ASC produces at least for performance critical loops. This is why I started writing AS3C but it is basically a stupid idea to try optimizing on a byte code level but it integrated very well into my daily workflow.
The Flash Player could work much faster if we could access certain things with ActionScript. The lookup of a class can be stored in a local variable for instance which gives a significat speed boost. haXe already allows you to do that via var m = Math. Why do we have to wait for such things so long?

Thoughts

Now after writing about Alchemy, ActionScript and the ASC I think that you get the idea. What we need is a compiler that works. And once the new Alchemy opcodes are supported in the ASC it will be much more fun to write your lightning fast ActionScript code instead of converting C libraries. We need a mature language which allows us to work in professional large-scale environments and which allows the compiler to produce a good output.
Libraries which are a community-darling could become even better, easier to use and faster in execution speed which would result in better user experiences.
I would like to see a faster and more innovative development of ActionScript and the ASC. But then there are the politics and all the bussiness stuff so we will have to wait probably for ages until we get what we need. Right now we can use alternatives such as haXe. And the good thing is that other companies do not sleep as well. I would love to write C# instead of ActionScript but Silverlight just sucks. Maybe someone could write a C#->LLVM->ActionScript->SWF compiler toolkit?

PixelBender Outline is working with FDT only. Just grab the JAR file and place it into your Eclipse plugins folder (…/Eclipse/plugins/) to install it. After restarting Eclipse you can open the view using Window->Show View->Other->PixelBender->PixelBender Outline.

The outline is updated everytime you select a *.pbj file in the Flash Explorer.

• Download

AS3V is basically a tool that can check for syntactical correctness. It will do with your code what a compiler does but it keeps formatting metadata. That way it can generate an XML skeleton of your code and apply rules on the XML.

You can have a look at the first output AS3V produces here. The original source code is included in the comment at the top.

As you can see it will be possible to force coding standards for instance and to warn if code could be optimized. I hope that it will be possible to have a first version of a stable parser soon. ANTLR is still giving me a lot of trouble but it is the first time for me working with such a tool as well.

• http://www.joa-ebert.com/swfs/lectures/fotb08
• http://www.joa-ebert.com/files/swf/lectures/fotb08.swf

Here are the results for 1000 DisplayObjects with manipulation on their scaleX, scaleY, alpha, x, y and rotation properties. It is also very important that you take a look at the memory behaviour. A click starts the tweening.

• Hobnox
• GTween
• TweenLite

As you can see it makes a lot of sense to stay type-safe and to manage the memory you are using on your own.
I will talk about those concepts in my next session AudioTool’s Private Parts in Brighton and Boston. Since optimization on a code level is trivial the main performance boost is achieved by re-thinking algorithms, structures and concepts. I guess one of the most interesting topics will be the optimization of our cable solver which was running O(n^2) (really!), then O(n(n+1)/2) and could be minimized to a rare O(n(n-4)/2) worst-case.

If you know PixelBender, than you know that you can not create loops. What you could do is unroll all constant loops with a fixed length. If you know simple convolution filters like a blur, you know that you need an xy-loop and you know it should not be possible with PixelBender at all. Let me prove you wrong and have a look at dynamic loop unrolling with PixelBender (be careful with high values!).

I built a library to assemble and disassemble PixelBender kernels at runtime. I wrapped it also in a high level API so basically you can create a new Kernel by doing var kernel: Kernel = new Kernel();. Then when you need your shader as a ByteArray you simple call kernel.compile(). There are still some glitches here and there but I hope that I can release the source code pretty soon.

I figured out that it is definitly not easy to handle the mxmlc or compc in a way we needed it to be. The point is that we have for each plugin three modules A, B, C and a library D. A depends on D. B depends on A and D. C depends on A, B and D. Now to make it even more complex we have A, B, C and D in one project so that developing in this environment stays still simple. The solution to compile all modules in a way we need them to be was first generating unique entry points for the SWF files because we do not want to work with SWC libraries (or the SWF inside the SWC package). After having the unique entry points we compile D (and its dependencies as well) without including the source of A, B or C. Afterwards we can compile A. But B is dependent on A so we have to compile a SWF for A and then a SWC for A so B can link to A as an external library. C is also analog to this but needs B as a library as well. In the end it was 5am and I was happy to have the complete build working which generates a lot of SWF files and SWC libraries.

Since it is very hard to configure Ant for this (I was using XSL in the beginning for automated code generation) I started developing my own Ant tasks. I have to say that it is really simple and saved us hours of work. We have now one single XML file containing the dependencies and the Ant task will do the rest (code generation, compile tasks, etc.).

FDT is implementing it also in a nice way. So for instance if you have a Vector.<int> named vec and you write var sprite: Sprite = vec[0] this will result in an error because FDT knows that your Vector was typed int.

There are still some glitches here and there but I hope this will be fixed soon as well.

By the way I am using the beta site to update my FDT — so if you want to have Vector support too you should use the beta update site.

Due to the fact that I do not have much free time to continue developing AS3C I think it is the right time to release the source-code on the one hand and to let people experiment with it on the other hand.

You can either download the sources and build AS3C manually (you will need zlib.net) or download a binary from trunk/bin/.

When using AS3C you will need the ActionScript from the SVN. Remember that you write real ActionScript code which gets translated by AS3C. There is also one undocumented and very experimental feature existing. If you run as3c.exe -optimize main.swf you could get some speed improvements if you have heavy loops using the Math class. But it could also destroy the SWF so do not forget to make a backup :o)

• AS3C on GoogleCode