Step #1: Parsing PixelBender data

PixelBender compiles down to PBJ which is a two-address based bytecode format. Basically it contains instructions like this one which I have taken from Nicolas’ blog:

  sampleNearest f2, text0[f0.rg]
  sampleNearest f3, text1[f0.rg]
  mov f4, f2
  mul f4, f3
  mov f1, f4

Those PBJ files are parsed by Apparat into high level structures. This happens basically the same way I do it for ActionScript bytecode. Thanks to the work of Nicolas Cannasse this was a walk in the park.

Step #2: Optimizing PixelBender bytecode

The PixelBender compiler does not produce optimal code as noted by Nicolas. You can see it in the code I just posted. Or you can simply compile and empty shader and take a look at the disassembled output. Apparat performs only two very simple optimizations for PixelBender which are written especially for PBJ. Thos include copy propagation and dead code elimination. The compiler does create code which is not bad thanks to LLVM but there are unnecessary copies going on which are removed by Apparat.

Step #3: Loop detection

PixelBender still has no loops since its initial introduction in October 1st in 2007. Tomorrow it is October 1st in 2010. Go figure. So since there are no loops the only way people write loops in PixelBender is basically by copy and paste or using preprocessor directives. With JITB I want to send real shaders to the graphics card and I do want to make use of loops. So I wrote a PBJ bytecode based copy and paste detector which is based on ideas of the PMD copy and paste detector. When a segment is detected in a sequence I convert it to a loop. That way I can reduce the amount of data I have to send to the graphics card. This is very important since in the case of the demos I showed at FOTB they contained around 50.000 instructions. And my cheap graphics card was complaining that there are too many instrucitons. With the loop conversion this number drastically decreased.

Step #4: Converting PixelBender bytecode to GLSL

It is very easy to convert PixelBender to GLSL. The two address code format can be translated with some minor tweaks to GLSL with a very simple mapping. However this is not really optimal since you have only a two address code. When exporting to GLSL I perform one additional step which expands the two address based code to a three address based code. That way I do not emit a lot of unnecessary copies again.

That is basically the recipie to convert PixelBender PBJ files to GLSL shaders that perform well on current graphics card hardware. There are additional steps involved to get OpenGL shaders to work like PixelBender in Flash but they are only minor tweaks. I have no comparable metrics right now. However I will create a JITB applet version so you can have a look for yourself soon.

As you might know I have done a couple of different tools for PixelBender already.

When I switched to Linux a while ago I had to discover that there is no compiler for PixelBender available and also no PixelBender toolkit. Even PBDT does not work on Linux since it requires a compiler.

In the current state Apparat can read and write PBJ files. I already implemented the format two years ago for the outline view plug-in. But this time Nicolas Cannasse’s great work on the hxformat library was my reference.

I will add two optimizations specific for PBJ files. TAAS will not be used here since it is only an unnecessary overhead. You do not have to crack a nut with a sledgehammer. The PBJ format is already register based and has certain useful invariants.

To cleanup some mistakes of the PixelBender compiler I will add:

• Copy propagation
• Dead Code Elimination

The compiler is not bad. Compared to the ASC it is a beauty of technology that makes use of the LLVM. However I think there must be some mistake in the code since unnecessary registers are used (Note: I doubt this is LLVMs fault but more the way how methods like sampleNearest are bound by Adobe). This can be solved with the simple optimizations I will add.

I will add some more things as well like a GLSL conversion or a DSL to write PixelBender kernels in Scala. If we get lucky this will also lead to a true cross platform opensource compiler for the PixelBender language. I think I will name that compiler Hydra :P

The second is about scopes. If you redefine a variable in a different scope the refactoring is not correct. But I guess 99% of the developers will not get in touch with this issue.
In this example i would be refactored in the outer and inner scope which is not correct.

void evaluatePixel()
{
        int i;

        {
        	int i;
        	i;
        }

        i;

        out = sampleNearest(src,outCoord());
}

I also updated the PBDT plug-in to get rid of some errors. I did not include all feature requests yet because the FFK in Cologne is currently eating up all my time.

Features of the PBDT plug-in include code completion, syntax highlightning and some other nifty features. It can be integrated into existing projects in FDT or FlexBuilder. Let me know what you think and please report any bugs or issues back to me.

A special thanks goes to Arne Deutsch from the FDT team for helping me a lot and answering all my annoying questions.

• PBDT

But there is still a big problem when it comes to dealing with image data in Flash. I have currently different strategies with their pro’s and con’s but I can not decide for myself which one makes the most sense.

1. BitmapData

   When you are using only one BitmapData to represent an image you have of course access to all the native methods. Life will be a lot easier. But there are certain key issues with the BitmapData. The precision for each channel is limited to 8bit and the main problem is that you will always have to deal with pre-multiplied alpha. If I want to have a high-quality image processing library this is unacceptable.

2. Vector.<uint>

   A vector filled with unsigned integers makes sense because this is what you get from a BitmapData and what you can use to set the contents of a BitmapData but working all the times with one integer value and having to do the bit-hassle is really annoying.
   The main problem with a Vector of unsigned integers is simply that PixelBender does only accept a Vector of Number values with the length of width * height * 4. So everytime you want to use PixelBender you would have to convert your vector. And also remember that you have only a precision of 8bit per channel. Instead of using a Vector.<uint> I could also use a BitmapData — the only thing I gain from the vector is that I do not have to deal with pre-multiplied alpha.

3. Vector.<Number>

   A vector of normalized Number values makes a lot of sense. You are able to use PixelBender with this kind of Vector without having to convert anything. The only problem occurrs when you want to convert your vector into a BitmapData — but this problem can be solved pretty easy with a PixelBender identity shader. This is also what I used to convert a BitmapData into a vector of Number values since BitmapData.getVector returns only a vector of unsigned integers.

   Now all of this does not sound that bad. But there are a lot of problems actually. First of all every time you want to get/set a pixel you have to access four fields of a vector which is way slower than accessing only one field. And the main problem I have is the poor PixelBender support. First of all every time I start a ShaderJob with a vector of fixed length as the target I get an exception that the ShaderJob.start() wants to change the length of my vector. What the fuck? If my Vector has a fixed length of width * height * 4 I do not see why the ShaderJob should change this at all.

   Another problem is PixelBender itself. In order to rebuild a method like BitmapData.fillRect() you have to visit every channel for every pixel if the rectangle is the full image size. Using ActionScript for this task is out of the question because it will be way to slow. So I thought I write a fillRect-Shader. No problem. The shader works very well in the PixelBender Toolkit but I get only garbage in the Flash Player. Now I am really uncertain if I should continue with this approach since even simple tasks fail. If my target is a BitmapData it works very well by the way. So if I want to fill a rectangle with PixelBender and I have only a Vector.<Number> I have to create a BitmapData with equal size to use it as the target with the ShaderJob and then use an identity shader to convert the BitmapData back into a Vector.<Number>. And once we are using a BitmapData we have again lost the precision and we have pre-multiplied alpha. This would not be a problem if PixelBender would work with Vector.<Number> as good as with a BitmapData.

4. Vector.<RGBA> and single-linked list

   Having a Vector.<RGBA> in combination with a single linked list is by far the best what you can get in terms of speed. We use this in our audio engine as well. You need to access every field only once and you can iterate over the single-linked list very fast. There are only two major issues. In order to use PixelBender you have to convert this structure which has the length width * height into a Vector.<Number> of the length width * height * 4 and you will have the same problems as described above. If you want to have a BitmapData representation you will have to convert the Vector manually to a BitmapData which will also cost you a lot.

My main problem is that I do not like any of those possible approaches. In the old ImageProcessing library I used a BitmapData for each channel but I would not do that again. The main problem is that Adobe has released a very buggy version of the PixelBender run-time in Flash. I would criticize also that we have so many different formats for pixel data which do not fit together. Sometimes you need a Vector.<uint>, sometimes you need a Vector.<Number> — sometimes the endian is flipped (Adobe Alchemy) and working with the BitmapData only — which sounds reasonable — is a pain because you will never get around pre-multiplied alpha. I would love to have a primitive RGBA data type for image processing which I could feed into PixelBender (color4) and which I could get and set to a BitmapData as a Vector.<RGBA>. If those RGBA elements would be also a single-linked list — even better.

PixelBender Outline is working with FDT only. Just grab the JAR file and place it into your Eclipse plugins folder (…/Eclipse/plugins/) to install it. After restarting Eclipse you can open the view using Window->Show View->Other->PixelBender->PixelBender Outline.

The outline is updated everytime you select a *.pbj file in the Flash Explorer.

• Download

The user input is currently only a Math.random() but I think you will get the idea. I always put the output of the shader back into it and get a nice wave effect.

• astro_water.zip

Another interesting part will be the open discussion “3D in Flash: Now and the Future” which Ralph Hauwert will moderate.

I think it will be an awesome event — hope to see you soon!

A new year, a new Flashforum Conference. The organizers Sascha and Marc invite you from 2nd to the 5th of June to Cologne. It is my third time at the Flashforum Conference and I think it will be a great event just like the last two years. This time the conference will be also a little bit more workshop-based with two extra days (pre and post conference) full of workshops. I will have one session about Hydra like in Amsterdam and another one about my inline bytecode compiler and optimizer. Be prepared — it will definitly rock.

Probably you noticed that I was working with Hydra a lot in the past. Everyone who worked with BitmapData will love Hydra and I have the honor to introduce you into the world of shaders. And even better: It is not only about Flash since Hydra filters work with AfterEffects too.

Working with Hydra for some days gave me a better knowledge of how things actually work and I thought it might be worth trying something that is a little bit harder. It is very funny to implement a complex filter on the one hand and keeping it Flash compatible on the other hand. You will see a lot of this in the code.

I really enjoyed writing this one. It helped me again a lot and I think you might learn a lot from the source code as well. By the way I started first with a k of 4 but that was crashing Hydra all the time. Might have looked better but 2 is also ok. Have fun!

• kuwahara.hydra

Thanks to Zeh for poiting out that a short if is possible and succeeds in the Hydra compiler with Flash warnings enabled. So you can use them only if your values are constant but the use of this is already fantastic. See this nice “technodots” filter which shoul be 100% Flash compatible if we can trust the compiler. It works because all parameters and function results have been set to variables before doing the if.

I think Adobe showed this filter during the keynote. Could not find sources for it and my internet is at home to crappy in order to watch the video again but it was something like this if I remember correct.

Update: To complete the list of filters they showed during the keynote here is a cheap twirl as well.

• technodots.hydra
• twirl.hydra

I am not so sure about the spherize though. I have one if-statement inside and they are not allowed if you use Flash. Funny new problems will pop up but I have a good feeling and one thing is for sure: the syntax and editor are already better than what we have with AS3/FlexBuilder! Highlightning of matching brackets, no stupid function and no var keywords. Finally! Can I have that with AS3 please?

• Oldschool – Simple oldschool plasma effect. You have to use a color palette with this.
• Ripple filter – Sine distortion for an image.
• Sineplasma – Sine plasma with adjustable levels, frequency and amplitude.
• Spherize filter – Spherical displacement map with adjustable center, radius and refraction strength.
• Structure – Filter with adjustable 3×3 structuring function.
• Warp – I wanted to do an affine transformation until I found out it does not work with the way hydra works. I will try again.
• Gamma correction – Look how simple it is!
• Pixelate – Could not be easier than that.
• Sepia – I hate those ColorMatrixFilter sepia effects. Now this one is fast and has nice quality using Y-transform.
• Quantization – A color quantization filter.