Tag Archive for 'imageprocessing'

Flash And Image Processing

During the last weeks I have thought about a completly new version of the image processing library. I have done lots of scribbling about what kind of features I want to have and which would be really cool.

But there is still a big problem when it comes to dealing with image data in Flash. I have currently different strategies with their pro’s and con’s but I can not decide for myself which one makes the most sense.

  1. BitmapData

    When you are using only one BitmapData to represent an image you have of course access to all the native methods. Life will be a lot easier. But there are certain key issues with the BitmapData. The precision for each channel is limited to 8bit and the main problem is that you will always have to deal with pre-multiplied alpha. If I want to have a high-quality image processing library this is unacceptable.

  2. Vector.<uint>

    A vector filled with unsigned integers makes sense because this is what you get from a BitmapData and what you can use to set the contents of a BitmapData but working all the times with one integer value and having to do the bit-hassle is really annoying.
    The main problem with a Vector of unsigned integers is simply that PixelBender does only accept a Vector of Number values with the length of width * height * 4. So everytime you want to use PixelBender you would have to convert your vector. And also remember that you have only a precision of 8bit per channel. Instead of using a Vector.<uint> I could also use a BitmapData — the only thing I gain from the vector is that I do not have to deal with pre-multiplied alpha.

  3. Vector.<Number>

    A vector of normalized Number values makes a lot of sense. You are able to use PixelBender with this kind of Vector without having to convert anything. The only problem occurrs when you want to convert your vector into a BitmapData — but this problem can be solved pretty easy with a PixelBender identity shader. This is also what I used to convert a BitmapData into a vector of Number values since BitmapData.getVector returns only a vector of unsigned integers.

    Now all of this does not sound that bad. But there are a lot of problems actually. First of all every time you want to get/set a pixel you have to access four fields of a vector which is way slower than accessing only one field. And the main problem I have is the poor PixelBender support. First of all every time I start a ShaderJob with a vector of fixed length as the target I get an exception that the ShaderJob.start() wants to change the length of my vector. What the fuck? If my Vector has a fixed length of width * height * 4 I do not see why the ShaderJob should change this at all.

    Another problem is PixelBender itself. In order to rebuild a method like BitmapData.fillRect() you have to visit every channel for every pixel if the rectangle is the full image size. Using ActionScript for this task is out of the question because it will be way to slow. So I thought I write a fillRect-Shader. No problem. The shader works very well in the PixelBender Toolkit but I get only garbage in the Flash Player. Now I am really uncertain if I should continue with this approach since even simple tasks fail. If my target is a BitmapData it works very well by the way. So if I want to fill a rectangle with PixelBender and I have only a Vector.<Number> I have to create a BitmapData with equal size to use it as the target with the ShaderJob and then use an identity shader to convert the BitmapData back into a Vector.<Number>. And once we are using a BitmapData we have again lost the precision and we have pre-multiplied alpha. This would not be a problem if PixelBender would work with Vector.<Number> as good as with a BitmapData.

  4. Vector.<RGBA> and single-linked list

    Having a Vector.<RGBA> in combination with a single linked list is by far the best what you can get in terms of speed. We use this in our audio engine as well. You need to access every field only once and you can iterate over the single-linked list very fast. There are only two major issues. In order to use PixelBender you have to convert this structure which has the length width * height into a Vector.<Number> of the length width * height * 4 and you will have the same problems as described above. If you want to have a BitmapData representation you will have to convert the Vector manually to a BitmapData which will also cost you a lot.

My main problem is that I do not like any of those possible approaches. In the old ImageProcessing library I used a BitmapData for each channel but I would not do that again. The main problem is that Adobe has released a very buggy version of the PixelBender run-time in Flash. I would criticize also that we have so many different formats for pixel data which do not fit together. Sometimes you need a Vector.<uint>, sometimes you need a Vector.<Number> — sometimes the endian is flipped (Adobe Alchemy) and working with the BitmapData only — which sounds reasonable — is a pain because you will never get around pre-multiplied alpha. I would love to have a primitive RGBA data type for image processing which I could feed into PixelBender (color4) and which I could get and set to a BitmapData as a Vector.<RGBA>. If those RGBA elements would be also a single-linked list — even better.

Online Photoshop Preview

Check out the post from John Nack or take a look at the screenshot directly. I am interested in what they offer. Actually if it is only the basic Flash filters being used this will not be anything new. Picnik and Fauxto are doing this already. We will see when they release it.

Optimized Seam Carving

Mario Klingemann did an awesome job on the seam carving demo I posted this weekend. He removed a lot of dirty stuff from the code and optimized it very well. You can get a full explanation on his blog.

As Mario mentions we are looking also in a technique that allows us to expand images. At the end the code is still not organized at all. A lot of performance can be gained with appropriate memory management. We keep you posted.

By the way there are still one or two tickets left for our rent an idea workshop! Last chance to have André, Mario and me for one day introducing you into this kind of stuff.

Content-aware image resizing

There is currently a lot of hype around this paper by Dr. Ariel Shamir and Dr. Shai Avidan. Honestly — it really kicks ass. This is a realy nice idea and I can absolutely understand Adobe just hired one of them direclty. Now there is a online demo of this available. Unfortunately it misses some source-code to look into and possibly optimize it.

So I did my own version. Basically this is a slim-version of the original technique. Image is converted to grayscale and a convolution matrix is used to find the pixel’s energy levels. Then the seams are created as the paper describes it … sort of :-). At the end the seam with the lowest energy is picked and removed.

The code is very ugly, not documented at all and not optimized. But it is also saturday night and I should be out drinking a beer already. You can press a key in the online demo to change the scale direction or click to reset it. If you can improve something do not be shy and post it.

Update: Here is another demo that shows the effect even better.

Update 2: Make sure to follow the joint venture from Mario and me.

Analyzing bitmaps bigger than 512×512

Remember I promised to release some paper on performance optimizations for the Flash Player? Well currently there is just to much to do in order to get it done. But I do not want to hide this from you any longer.

Analyzing big BitmapData objects is a pain. For a nice contrast correction, world luminance or any other histogram based filter you need to know about all the pixels (if you want to make it accurate).

On my way from Germany to France a month ago I was thinking about this problem and I found something that is strangely enough faster to read a whole picture — but only for big pictures. For smaller ones fortunately it does not matter that much since the time you have to spend on reading them is around 30ms.

So basically I read through a BitmapData like this:
[code]var x: int;
var y: int;

for (;x < width; ++x)
for (y = 0; y < height; ++y)
color = bmp.getPixel( x, y );[/code]

I guess it is very common to do that. Now this goes row by row (or column by column) through your BitmapData. Now just for fun I was testing what happens if you cycle through your BitmapData without ever resetting x and y. And voilá it is faster. On a 2048x2048 BitmapData it is 336ms vs 739ms. That is a nice performance increase. But funny enough it is slower for smaller bitmaps.

Here are the results (cycling vs. for-loop):

Iterations Cycle[ms] Simple for[ms]
128x128 2.7 2.7
128x128 11.1 10.3
512x512 29.7 28.3
1024x1024 95.4 106.3
2048x2048 355.8 738.6

And here is the code for the faster version:
[code]var y: int;
var x: int;
var m: Boolean = true;

while (true)
{
color = bitmapData.getPixel(m ? x++ : --x, y);

if (x == width || x == 0)
{
if (y++ == height)
break;
m = !m;
}
}[/code]

ImageProcessing will go open source

With the release of popforge there was another announcement as well. The Google Code page says that a future project is the ImageProcessing library.

Probably the current version — wich I am not happy with actually — will go there and a better version will come soon supporting asynchronus filter execution for instance. I will not give any date for now but expect a release in the next couple of weeks.

Lorenz spectrum

LorenzSpectrum

A different approach for a spectrum visualizer. This one is based on a lorenz attractor and I really like how fast everything is calculated and gives a feeling of speed. This time I also had to decrease the amount of particles I render. Instead of 64k (which was still running smooth) I decided to use just 8k because it looks better.

What I like about it is that it looks complex but is very simple to do. Based on the spectrum I just change the parameters of the lorenz attractor and that is it. A little bit of paletteMap, BlurFilter, Matrix and ColorMatrix of course as well but it is only about a 100 lines of code.

Adobe Photoshop online

Three days ago there was some news about an online version of Adobe Photoshop. Today I read on a German news site that they will probably use Flex. I am so excited about that news and I am looking forward to try their product. Of course I have a special interest in this because I developed my own library and now the big boys do it.

It will also be very interesting to see what they will do about some issues. If you want to edit a picture (like 6megapixel picture) it will be a big fight against stuff like the maximum execution time. Probably they will use an approach to filter an image over more than one frame. So it would be possible – just like Photoshop does – to see a progress of that thing. But even in that case such a filter would take you at least half a minute of waiting to complete. Maybe they will start working on some performance issues of the Flash player as well..?!

Anyways I can’t wait to see this in action!

Update: I am quiet sure that we will see two techniques here. One will definitly be connected to Event.ENTER_FRAME for extensive x,y-loops and a lot of getPixel()/setPixel() calls. The second method is also connected to Event.ENTER_FRAME and goes rectangle by rectangle for stuff like paletteMap(). Just speculations. We will see more in six months.

Some updates

I added two the Imageprocessing Library some new blurs. RadialBlur, ZoomBlur and GeometricDisplacedBlur. The GeometricDisplacedBlur is very nice, because you can use any GeometricMap with this filter.

I also added a RedEyeRemoval and made some updates to the Color class. Some other filters have been updated as well. I will focus now on some examples and more GeometricMap objects. I am also thinking about filters for HSV color correction and this stuff. But that is not very hard since it is only the ColorMatrix that has to be extended. I am also trying to find good ways for texture synthesis (that will not freeze Flash for 10 seconds).

If someone has any idea for filters that are missing just post it here please.

Render, noise and morphological filters

Dilation

I had a lot of stuff on my mind the last days (besides the Wiimote). Check the documentation for nice pre-views. Here are the resuts:

de.popforge.imageprocessing.filters.*

  • Closing
  • Dilation
  • Erosion
  • Opening
  • Outline
  • TopHat
  • Curves
  • Blobs
  • CellGenerator
  • Corona
  • FractalPlasma
  • MakeTileable
  • Mandelbrot
  • OldschoolPlasma
  • Tile
  • AverageCut
  • MedianCut

de.popforge.imageprocessing.geom.*

  • Curve
  • StructuringFunction