LZMA compression in Apparat RC5

Matryoshka avec moustache

I have released Apparat RC5 at GoogleCode. It contains a really cool feature which is called LZMA compression.

Reducer has advanced a lot during the last couple of weeks. It is now also a strong SWF compression tool even if you do not have any PNG files it can compress. You may ask: “What is that Matryoshka doing there? And why the hell the top hat?” The top hat: I do not know. The Matryoshka: I can explain.

The Flash Player does not understand LZMA. SWF files are compressed using good old DEFLATE. So what happens? Apparat extracts your original SWF. It compresses it again using the LZMA algorithm. The compressed SWF is injected into another SWF that contains an LZMA decoder. Size, background color, frame rate, etc. get adjusted. Finally you get a new SWF that contains your old SWF and a decoder to extract it at runtime. The overhead of the decoder is currently at around 5kb and I hope I can get it even smaller.

When you open that SWF with the Flash Player it will extract your original file and load it. Another nice feature is that I created different versions of the runtime decoder. One is using a classic preloader which is great if your SWF is a little bit bigger. And hey: it is a preloader for free so you do not have to deal with the [Frame] hassle. But here is the catch. We at audiotool.com always write our SWF files in the same style and I can just hope you do the same or use the great InitInjector by Valentin Simonov.

Your main SWF class or the so called DocumentClass must make sure that a stage is available before accessing it. This is really easy:

public function Main() {
  addEventListener(Event.ADDED_TO_STAGE, init)
  if(null != stage) init()
}

private function init(event: Event = null): void {
  removeEventListener(Event.ADDED_TO_STAGE, init)
  // your original constructor code goes here ...
}

Stick to that rule or InitInjector can automate it for you. Otherwise you will get a runtime exception that the stage is null. So this is one new feature. The other one is actually pretty standard. If you compile and link against a SWC file all classes will come in their own ABC file. Each ABC file has a constant pool. So if you link against 1000 classes you get 1000 constant pools. Reducer can merge all those files into a single one with some minor exceptions. It can also sort the constant pool so that constants which are used frequently consume less bytes when accessed. The funk-as3 test runner which links against FlexUnit and the Flex framework is only loosing 3.01% of its weight with the old Reducer. The new version reduces it by 17.81% already thanks to the ABC merging. Combine that with some LZMA love and we get 35.34%.

Besides you can call the LZMA compression as often as you want for some basic obfuscation. The LZMA compression alone is already a (weak) obfuscation of your bytecode.

Last but not least: Good news everyone! Scala 2.8 arrived so this is the last time you will have to update it for a while.

You can download the latest Apparat version at GoogleCode. Scala 2.8.0 is required.

31 Comments

  1. Posted Jul 16, 2010 at 12:14 am | Permalink

    Im gonna have a go with a AS3 flash file that is running very slow. Lets see if it makes a difference and how compressed it gets!

    Thanks Joa, you are a cool!

  2. Jonathan Hardie
    Posted Jul 16, 2010 at 4:31 pm | Permalink

    I’m trying to run Reducer via the command line on OS X and I get this the following message – would appreciate any suggestions:
    java.lang.NoClassDefFoundError: apparat/log/SimpleLog
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$findClass(ScalaClassLoader.scala:86)
    at scala.tools.nsc.util.ScalaClassLo…

    Really looking forward to trying this improved compression, thanks for the hard work!

  3. Posted Jul 16, 2010 at 5:59 pm | Permalink

    Hi,

    sounds like a problem with the raped OS X shell.
    Can you please try if this command works for you:

    scala -cp `pwd`/* apparat.tools.reducer.Reducer

  4. Jonathan Hardie
    Posted Jul 17, 2010 at 11:15 am | Permalink

    Here’s what I get when I run that command – not sure what the indended output is supposed to be: http://pastie.org/1048373

  5. Posted Jul 17, 2010 at 12:23 pm | Permalink

    What does “java -version” say? 1.6 or 1.5?

  6. Florian
    Posted Jul 18, 2010 at 11:58 am | Permalink

    You’re awesome :) And so is apparat. Keep up the good work.

  7. Jonathan Hardie
    Posted Jul 19, 2010 at 10:53 am | Permalink

    java version “1.6.0_20”
    Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-10M3065)
    Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)

  8. Posted Jul 19, 2010 at 11:50 am | Permalink

    Oh sorry, the command would have been

    scala -cp `pwd`/\* apparat.tools.reducer.Reducer

  9. Posted Jul 19, 2010 at 1:31 pm | Permalink

    I’m getting good results using Apparat Reducer, but have one question:

    Is it somehow possible to direct Apparat Reducer to not use LZMA on certain parts of one .SWF? Or, to have an initializer function that doesn’t get compressed using LZMA, but only gets compressed normally using deflate? Perhaps using metadata?

  10. Posted Jul 19, 2010 at 2:18 pm | Permalink

    Phew. Well this already happens: the decoder is not LZMA compressed. What is your usecase?

  11. Posted Jul 19, 2010 at 2:28 pm | Permalink

    My use case is quite eccentric I think: on the server, I’m unpacking an .SWF, injecting a code using search/replace, and repacking again. I haven’t tested yet, but I think the search/replace doesn’t work anymore, because it’s already LZMA compressed.

  12. Posted Jul 21, 2010 at 2:57 pm | Permalink

    That is correct. But you could use Apparat on your Server for the Search/Replace job, and let it simply pack the Swf to LZMA afterwards.

  13. michael
    Posted Jul 23, 2010 at 7:38 pm | Permalink

    This is a really interesting program, I’ve been experimenting with it quite a bit.

    I have a specific question relating to working with the swcs: I simply replaced all relevant Math calls in away3d with FastMath calls… somewhat surprisingly, i don’t see any difference in performance between the raw and post-tdsi version (i tried variations of all parameters)?

  14. Posted Jul 24, 2010 at 12:32 pm | Permalink

    Hi Michael,

    regarding Away3D: You need to use inline expansion to make use of the FastMath. However I do not think that the Math is a real bottleneck here. The main problem is the drawing. And that cannot be fixed by changing the ActionScript bytecode.

    Other people that make extensive use of 1/Math.sqrt reported something like 50% speed increase in their project using FastMath.rsqrt

  15. Posted Jul 29, 2010 at 2:32 pm | Permalink

    Great work, as usual :)

    Will I need to build a custom version of Apparat to change the decoder/wrapper injected by Reducer when using LZMA, or can it be specified in the build somehow? (it would be great to be able to specify the wrapper code in the Reducer ant-task)

    I can see myself also needing a Flash 9 compatible version of the decoder/wrapper – so types will have to go :(

  16. Posted Jul 29, 2010 at 2:37 pm | Permalink

    * – so Vector types will have to go :(

  17. Posted Jul 29, 2010 at 5:49 pm | Permalink

    You would just have to create another Matryoshka that does not use the apparat-lzma-decoder library and instead your own. Or you could also simply change the lzma-decoder library and create an Apparat build.

    That is actually quite simple. Download Apparat and do “mvn package assembly:assembly” to generate a ZIP package for you.

  18. Posted Jul 29, 2010 at 6:12 pm | Permalink

    Thanks, will look into that.

  19. Posted Jul 29, 2010 at 11:28 pm | Permalink

    Ah, got it working :)
    In the end I just replaced all Vectors in the LZMA decoder/Matroyska class with Arrays and rebuilt… works a treat.

    Didn’t realise this was such a huge project until I needed to build it, amazing work.

    (Oh, and in case anyone else tries it, the command to build+package the custom Apparat was actually: “mvn package assembly:assembly”)

  20. Posted Jul 29, 2010 at 11:58 pm | Permalink

    I am happy it was that easy for you. However, I had to resturcture the project a little bit today and it changes the assembly generation.

    It is still very simple. “mvn install” in the Apparat root, then go into apparat-assembly and run “mvn assembly:assembly”.

  21. Posted Jul 30, 2010 at 12:42 am | Permalink

    Oh right, I think I picked up those updates and lucked into running “mvn install” (by following the maven quickstart guide) before later running the assebly generation.

    Thanks again.

  22. Posted Aug 2, 2010 at 3:03 pm | Permalink

    I’ve written a little bit about my Apparat experiment here:
    http://blog.madebypi.co.uk/2010/08/02/apparat-a-custom-matryoshka/

  23. Posted Aug 31, 2010 at 6:01 pm | Permalink

    Hello Joa,

    I tried your Reducer on a 2mb swf with Apparat RC6, and I came up with a tiny 0.9% compression ratio. I guess the LZMA compression didn’t work as expected.

    How could I check if LZMA has been used during reduction ?

    By the way, thanks for your set of supa cool tools !

  24. Posted Aug 31, 2010 at 6:36 pm | Permalink

    You have to enable LZMA compression via the -l switch when running reducer.

  25. Posted Aug 31, 2010 at 7:05 pm | Permalink

    Effectively, it goes up to 3.85% reduction. You might add the -l switch to your README file for dummies like me.

    Thanks again

  26. Posted Sep 17, 2010 at 12:33 pm | Permalink

    Hi,

    Thanks for sharing all the great work !
    I got very good results with the LZMA option, but actually i’m stuck with the fact that i cannot access the loaded swf application methods cause of the Matroyska wrapper.
    If i understand good, i should extend the Matroyska with some methods to access the wrapped loader content ?
    What i’m exactly doing is loading a flex application in a flash wrapper. And i need this wrapper to communicate with the flex application main class (pass flashvars, execute methods on resize…)

  27. Posted Sep 19, 2010 at 1:18 pm | Permalink

    Eric, try looking up the class in the loaders application domain. I think its something like loader.contentLoaderInfo.applicationDomain.getDefinition(‘com.your.Class’)

  28. Posted Sep 20, 2010 at 3:49 pm | Permalink

    @joa thank you for your answer.
    I’ve been able to retrieve the flex application instance with this code :
    loader.contentLoaderInfo.applicationDomain.getDefinition(“mx.core.Application”).application

    What i achieve is compress my flex application from 613ko to 470ko, but now the initialization time has grown up a lot (something like from 1s to 2/2.5s), and that is very problematic for me as my application has to be fastest as possible to initiate (kind of a widget).
    Is there any benchs on the LZMA decompression times ?

  29. Posted Sep 25, 2010 at 10:51 am | Permalink

    Eric: There are no benchmarks yet but I know performance can be improved a lot regarding decompression time. However I will not be able to work on this before FOTB since I was very busy the last weeks.

  30. Posted Sep 28, 2010 at 5:00 pm | Permalink

    No problem Joa, I understand you got other priorities, and you have done already a lot of work for the community.
    Thank you for that !

  31. Amor Nguyen
    Posted Dec 3, 2010 at 6:11 pm | Permalink

    i don’t know how to decrompress a .7z file with Apparat in AS3. Can you give me an example code. Help me, please….

4 Trackbacks

  1. […] This post was mentioned on Twitter by Mario Klingemann, Joa Ebert, Ryan Christensen, Ryan Christensen, W. Schmidt-Sicherman and others. W. Schmidt-Sicherman said: RT @Quasimondo: SlimFast for Flash – @joa's LZMA compression makes all your swf files smaller: http://bit.ly/bAM7sw […]

  2. […] few weeks ago Joa Ebert (@joa) released an update to the Apparat Reducer, adding the option of LZMA compression to further reduce your SWF […]

  3. […] the Apparat framework, give your swf the extra speed via TurboDieselSportInjection and run the apparat reducer to compress your swf. First we need to make an instance of the ApparatToolkit, just like we did for […]

  4. […] the Apparat framework, give your swf the extra speed via TurboDieselSportInjection and run the apparat reducer to compress your swf. First we need to make an instance of the ApparatToolkit, just like we did for […]