Tag Archive for 'audiotool'

Project Hiddenwood

Published on Nov 11, 2011 in news, personal and projects. 15 Comments Tags: , , , , , , .

This years FOTB was special. At the end of my session I showed a sneak preview of project Hiddenwood. I demonstrated complete playback of Audiotool tracks on stage — in a browser. Now that does not sound too special…

But then again, the playback was done using JavaScript only and calculated in realtime.

Audiotool is a complex piece of software so you might ask how one could torture themselves by implementing it in JavaScript? We didn’t. Instead we started building our own vision of a cross-platform application framework a couple of months ago.

Introducing project Hiddenwood.

Hiddenwood is a collection of libraries and tools specifically designed to support different devices and platforms. The core libraries are the “driver layer” and always platform-specific with a platform-independent interface.
On top of that we provide a basic layer of libraries like our UI system, animation framework or managed collections which guarantee 0% garbage collection activity and have been battle-tested in Audiotool.

The framework is all about speed and consistency. The rendering pipeline is optimized for OpenGL and although we offer something similar to Flash’s display list a lot of features are not available because they would compromise the speed.

Speaking about speed: we are always interested in staying as native as possible on our target platform. So for the browser we emit JavaScript, for Android you will get the full DalvikVM performance and for the desktop you will get JVM performance. This approach has also another very important aspect. If you want to go platform-specific for certain features you can do that.
For instance if we want to render Audiotool songs on the server using a fork-join pool for our audio calculation this is possible and might not make sense on an Android device.

You write Java code and the supported platforms are native desktop applications, Android (minimum requirements are Gingerbread and OpenGL ES 2.0) and modern browsers. Now for browsers we even go one step further and support multiple options. That means if WebGL is not available we simply fallback to a normal canvas based render-engine. The same applies to some of the Android drivers.

iOS is of course important as well and we are actively researching the best option that will give us the most flexibility and performance.

We are currently working on two real applications built with Hiddenwood. So far it is a real pleasure to enjoy quick build times and simply test what you want on the desktop with great debugging capabilities. When you are ready you can try the same app on Android or in the browser — which might take a little bit longer to compile.

Because we see Hiddenwood as an application framework there are a lot of goodies built-in like a sprite-sheet based class generator. Think Image mixerBackground = Textures.mixer.background(); where mixer was the folder name and background the name of the file.

We believe that as a developer you really do not care about what kind of technology you are using and just want a great result. We also think that you should be able to reuse platform-independent code across multiple projects. However we do not want to take power away from the developer because if you know what you are doing: go for it.

Of course we are not the only ones with this idea. Nicolas Cannasse saw the signs years ago and invented haXe which gives you a comparable experience and Google released playN a couple of weeks ago which takes a similar approach (and requires another 25 committers :P).

But when we started Hiddenwood we wanted the Java tooling experience and playN was not public at that time. We also think that a game engine is not what you want to use for all kinds of applications. So we like to be able to give people the freedom to build their own game engine on top of Hiddenwood — and calculate physics in a different thread peut-être.
Speaking about threading: the only possible solution that works across all platforms is a shared-nothing architecture which we put in place. However if you write platform specific code you can use of course everything the platform offers and a lot of the Hiddenwood core libraries like the network- or cache-layer make use of multiple threads.

In the end what makes Hiddenwood special in my opinion is that we do not believe in write once run anywhere because that just does not make sense. The essence and philosophy behind Hiddenwood is to write platform-agnostic code using kickass-libraries and being able to reuse that. Audiotool on a tablet would look completely different from Audiotool running in a browser. And Audiotool on iOS would probably be also a little bit different from Audiotool on an Android device because there are simply different paradigms you should respect.

I hope that we can share more information with you soon. With the news of mobile Flash Player being deprecated and the ongoing demand for cross-platform development we have exciting times ahead of us. I am also super excited about the (beautiful <3) applications which we are going to release in the not so distant future.

Software Transactional Memory and Audiotool

Published on Feb 25, 2011 in as3, Flash, knowledge, personal and projects. 5 Comments Tags: , .

I have tweeted a while ago about the fact that Audiotool does heavily rely on a so called software transactional memory. In this blog post I want to discuss the implementation of our STM and why it is useful.

Wikipedia defines STM as:

In computer science, software transactional memory (STM) is a concurrency control mechanism analogous to database transactions for controlling access to shared memory in concurrent computing.

First of all when reading this you might think that an STM is not necessary since ActionScript does not support true concurrency and also the shared-nothing worker proposal will not allow you concurrent memory access. While this holds true you may not forget that concurrency becomes an issue when you have just access to a single core but on multiple machines.

A lesser known fact about Audiotool is also that we have a built-in system for real time collaboration since its beginning. Due to conceptional issues and other prioritized features we were never able to roll it out as a full feature but the technology base is there.

When it comes to multiplayer gaming and network latency you will always have to figure out a system which allows you to share a certain state across multiple machines. In some games the state is synchronized across a certain server. An example is an FPS like Quake3. The Audiotool is unlike Quake3 not a FPS and different conditions have to be met. And this is the reason why I have choosen to implement an STM with very optimistic concurrency control.

The basic idea of my system is to use atomic transactions with commit and revert operations. Those are handled by a transaction manager which performs the basic tasks of an STM. That means: execute all transactions until an error occurs, roll back and try again. Each transaction in the Audiotool contains a guard. The guard defines whether or not one should still execute the transaction.

Here is an example of why a guard is important and how all this stuff works. You have two Audiotool instances running that share the same state. We call them A and B. A and B are connected via P2P no an arbitrary network. This means exchanging messages introduces latency and we cannot guarantee that A is at the same state of B.

So what happens? t0 is committed by A, serialized and sent to B who is committing the same transaction. The same applies to t1. t2 is initiated by B, serialized and sent to A. Everything works fine until B commits t4. While t4 is being transferred to A, A created t5 and committed it. This means A never saw t4 when it committed t5 and B would continue with t6 ignoring t5. A will actually receive t4 after he commited t5 so the system will revert t5, commit t4, commit t5 if the guard does not prevent it, and both continue at t6.

To explain the guard: Imagine the t5 transaction includes modifications on a device in the Audiotool that is deleted by t4. In that case t5 will never be executed.

So what does this imply? You can guess that each modification of the Audiotool state is in fact handled by a transaction and we carefully have to pick the guards. E.g. if an editor is referenced by a transaction it is part of the guard. However this introduces a second issue: references. Sharing references of objects across a network is a tricky task. I am using “boxes” to solve this issue. A box can hold a value and it is guaranteed that it will hold the same value at the same state across the network. A box can be serialized and will be shared between commit and revert states. This means if tn creates an object which is referenced by tn+1 both will use the same box. This also means that I can safely revert tn+1 and then tn and commit them without having to worry that both still reference the same object. And this even works across the network. And in fact we can now also identify whether or not one transaction would conflict with a different one which is a very important condition for the actual implementation.

Right now you might have figured out that a transaction in the Audiotool has the following properties:

  • It is serializable so it can be transferred across the network.
  • All object references must be stored via “boxes”.
  • A guard is used to determine whether or not to commit a transaction.
  • Transactions can be committed and reverted.

A lot of stuff one has to reason about. But although we do not support live collaboration at the moment we are using the STM. Why? The reason is quite simple. The STM gives us history for free. Since everything is serializable via a very lightweight protocol we can take the last 500 transactions for instance and store them as a compressed ByteArray in memory instead of each as a single object with all its references.

Another very nice option is testing. A serializable history means that one simply has to press a super secret shortcut to dump the whole history and send it to the responsible developer who can reason about every single step that happened which ultimately led to the error. In development mode I usually commit, revert and commit each transaction. Then I also serialize and deserialize them and do the whole thing again.

Since transactions are known by the system we can also write a fuzzer that executes lots of crazy stuff. Sort of Android’s monkey mode. And ultimately I personally hope that we will get to the point where we will enable live collaboration in the Audiotool of course.

The last feature I implemented is the automatic transformation of arbitrary transactions to a compound transaction. You can guess what this feature actually means for the user :). In fact it is very complex and I implemented it with the STM using only a couple lines of code (of course that is just half the story …).

If you are interested in writing your own multi user application I think this is a very nice approach since collisions do not happen very often from my experience and the optimistic concurrency in a P2P environment achieves great results in terms of overall latency and feeling. The Google Wave protocol and their operational transform is also an interesting read.

Audiotool 1.0

Published on Mar 25, 2010 in Flash, news, personal and projects. 9 Comments Tags: .

We have finally released the first version of the Audiotool. I am very thankful for all the kind tweets and happy that our work is now available to the public.

However the current version is named Firestarter for a reason. What you see online is just the tip of the iceberg. I can not wait to start implementing all the cool features that will separate us from traditional software ;)

AudioTool Update With ToneMatrix

Published on Apr 17, 2009 in Flash, misc, personal and projects. 6 Comments Tags: , , .
AudioTool with Tonematrix

We have release a small update of the AudioTool and included André’s ToneMatrix. Some features that have been implemented in the last weeks are online as well. Removing an effect for instance will not destroy the cables. The desktop scales now without interrupting the audio output and various other performance tweaks have been made.

Graphs and a floating world

Published on Feb 16, 2009 in as3, Flash, personal and projects. 1 Comment Tags: , , , , .

When you are implementing a pathfinding algorithm you have usually some sort of defined map available. This is in most cases a tile map or a graph of nodes. If you have no nodes and no tiles you have two options: build a graph of nodes or put your smooth environment into a raster with a defined detail. Option two is usually faster but we have choosen number one for asthetic pleasure in the AudioTool. And we showed that building such a graph at runtime in Flash is something you can do if you implement clever optimizations.

You will always have problems without some kind of raster. Sometimes more, sometimes less. Usually you have to build your own context of how things are connected and how they behave in your environment. We were discussing a lot about the audio engine and the core structure behind. The most important part of the audio engine is probably the ability to build a linear chain of plugins which process signals that arrive in your speakers at the end of that chain. When we started this part we had a lot of optimizations in mind. Plugins that do not generate any audible output should not go into that chain etc. The code became quite complex and results in a lot of special case scenarios. Imagine a scope which is a pure visual element that shows the incomming waveform. This is a plugin which should not be kicked out of any chain but generates no audible output.

You can imagine that we used a graph for that kind of task. To be more specific: a directed cyclic graph since there is a direction of the flow and we are allowing feedbacks for audio signals. After some time of discussion I started researching for a better way of solving our signal chain. Because I beleive that a problem can be solved the easiest way on a pure mathematical basis I started with a plain graph — but this time a real graph. After some time I could produce the same output as our old solver but with less lines of code and we got rid of all the special cases.

Automatic Graph LayoutNow having a real graph is fun and I was not surprised that lots of people invested their brainpower in graph algorithms. For the desktop in the AudioTool I always wanted to implement some kind of automatic layout algorithm. My first idea was to build a spring between two editors connected by a cable. I thought that this could be too complex but after spending the last days with graphs this became a possible solution and there are already algorithms which do the same thing: build springs for connections and let the system solve itself. I did some tests in a sandbox already but hopefully I can implement this in the AudioTool before FITC Amsterdam. The graph you see in the picture is a result of the graph layout engine.

But working with graphs opened my eyes even more. I am currently the poor bugger that has to implement a layout engine. Implementing a layout engine is by the way the most ungrateful task I could think of. Layout Framework Anyways, what if you think about the dependencies in your layout engine as a graph structure? You have usually three cases when calculating a size: relative known size to “some kind of available size”, absolute known size, absolute unknown size. Those cases can be reduced to their most simple form on a piece of paper and a graph structure evolves. You can get rid of observers and listeners in such a system if a connection in the graph has the meaning of invoking the layout engine again on a subset of nodes. Those subsets could be optimized even more if you figure out the strongly connected components in the graph. I have to admit that I did not implement this system yet but it makes a lot of sense in theory currently… something which makes working on a layout engine interesting again ;)

Rasselbock loves the Lemur

Published on Feb 11, 2009 in experiments and Flash. 6 Comments Tags: , , , , , .

This is what you can get when you combine a Lemur with the AudioTool and map some controls to the Rasselbock. I really hope that we can release the Midi feature to the public soon because it is just so much fun!

AudioTool On A TouchScreen

Published on Jan 28, 2009 in experiments, Flash, personal and projects. 1 Comment Tags: , , .

Comming into the office and seeing a video like this makes me really happy and proud.

Tones

Published on Nov 20, 2008 in news and personal. 0 Comments Tags: , , , .

I just put up a site with all the AudioTool recordings that I think are quite interesting. It is way easier to share them like that instead of posting about every single track.

I will also put some more sets and other tracks there if I find the time. The site will update randomly.

AudioTool’s Private Parts Slides

Published on Oct 1, 2008 in as3, Flash and knowledge. 8 Comments Tags: , , , .

Here are the slides from my Flash On The Beach 2008 session “AudioTool’s Private Parts”. 2mb without a prelaoder. You will need Flash Player 10 to pass through to version detection or you can save the SWF directly to disk and watch it with Flash Player 9.

Compiling dependent SWF files

Published on Sep 5, 2008 in as3, Flash and projects. 4 Comments Tags: , , , , , , , , .

AudioTool Ant buildLast night I spent about 9 hours writing a build script for the AudioTool allowing us to compile each plugin into its own SWF container. I think it is the first time a Flash project is taking more than 5 minutes for me to compile completly.

I figured out that it is definitly not easy to handle the mxmlc or compc in a way we needed it to be. The point is that we have for each plugin three modules A, B, C and a library D. A depends on D. B depends on A and D. C depends on A, B and D. Now to make it even more complex we have A, B, C and D in one project so that developing in this environment stays still simple. The solution to compile all modules in a way we need them to be was first generating unique entry points for the SWF files because we do not want to work with SWC libraries (or the SWF inside the SWC package). After having the unique entry points we compile D (and its dependencies as well) without including the source of A, B or C. Afterwards we can compile A. But B is dependent on A so we have to compile a SWF for A and then a SWC for A so B can link to A as an external library. C is also analog to this but needs B as a library as well. In the end it was 5am and I was happy to have the complete build working which generates a lot of SWF files and SWC libraries.

Since it is very hard to configure Ant for this (I was using XSL in the beginning for automated code generation) I started developing my own Ant tasks. I have to say that it is really simple and saved us hours of work. We have now one single XML file containing the dependencies and the Ant task will do the rest (code generation, compile tasks, etc.).