I am definitly not good at choosing names for software projects. However TurboDieselSportInjection is a release of my experiments from yesterday. It is a spinoff from the whole framework and allows you to inline __bytecode and of course to use the new Memory API.
Hopefully you are kind enough to provide me with some feedback. I am especially interested in Exceptions that occur when reading or writing ABC files. Have fun!
Update: TDSI is now open source!
I have pointed that at several small swfs and keep getting the following error (same for each one)
java.lang.NullPointerException
at com.joa_ebert.abc.bytecode.permutations.PermutationChain.visit(PermutationChain.java:25)
at com.joa_ebert.abc.Method.accept(Method.java:59)
at com.joa_ebert.abc.Abc.accept(Abc.java:124)
at com.joa_ebert.tools.tdsi.TDSI.run(TDSI.java:163)
at com.joa_ebert.tools.tdsi.TDSI.visit(TDSI.java:206)
at com.joa_ebert.swf.tags.control.DoABCTag.accept(DoABCTag.java:23)
at com.joa_ebert.swf.Swf.accept(Swf.java:69)
at com.joa_ebert.tools.tdsi.TDSI.run(TDSI.java:105)
at com.joa_ebert.tools.ToolRunner.run(ToolRunner.java:47)
at com.joa_ebert.tools.tdsi.TDSI.main(TDSI.java:36)
Joa may I ask you which is the framework you are talking about in your post? I suppose it’s the Java framework you use to create the tdsi Jar file. It seems to be a large framework.
Conrad: Thank you very much. It should be fixed now.
Tek: The framework I am talking about is a spare time project of mine and currently in development. Indeed it has already grown pretty large.
Can you give a simple example where this leads to a performance increase? I’ve been trying to get a simple routine (Mandelbrot) faster, but I end up with something just a bit slower than a version not using Memory.
This is quite simple:
for( var i: int = 0; i < 1000000; ++i ) byteArray.writeByte( 0 );
vs.
for( var i: int = 0; i < 1000000; ++i ) Memory.writeByte( 0, i );
Or for instance instead of doing something like BitmapData.setPixel you can now write your data to a ByteArray and use BitmapData.setPixels. You could have done the same before using Vector. but Nicolas claims that the new opcodes are about 30% faster than Vector access.
can’t wait to play around with this when I get home! Looks like great work again-
Hi Joa,
Got this Flash player error:
VerifyError: Error #1045: Unsupported traits kind=14.
then…
ReferenceError: Error #1065: Variable PanoTest_img is not defined.
Guess its because i got [Embed] metas?
Very cool stuff though!
Thanks for the example, that is indeed quite simple and I see how that is faster :). But now I also get the point that it’s not just about replacing some assignments and reads with Memory.read/writeXXX, but also about finding a good purpose for it.
Tim: Yes, [Embed] is a problem which I will tackle today. Should not be too hard.
JW: You got it ;)
So I basically copied your framebuffer and wrote a simple noise function:
private function noise(e:Event):void {
var i:int = WIDTH * HEIGHT * 4;
while (i > 4) {
i -= 4;
Memory.writeInt(Math.random() * 16777215, i);
}
_buffer.position = 0;
_screen.lock();
_screen.setPixels(_screen.rect, _buffer);
_screen.unlock(_screen.rect);
}
I get 38fps with this script. But following your optimization tips, I should provide Memory.writeInt with an int:
var c:int = Math.random() * 16777215;
Memory.writeInt(c, i);
But now I get only 34fps? From what I understand of compilers (very little) I suspect some casting is going on, because when I:
Memory.writeFloat(Math.random() * 16777215, i);
I get 39.5fps(, and a slighter more blueish noise).
I thought maybe the extra variable c caused the delay, but when I then:
Memory.writeInt(int(Math.random() * 16777215), i);
TDSI crashes with:
[-] Unknown Memory call.
java.lang.RuntimeException: Unknown Memory call.
at com.joa_ebert.abc.bytecode.asbridge.MemoryInlineJob.interpret(MemoryInlineJob.java:155)
But this is great fun anyway :)
Thank you for the bug report. I will investigate in the crash.
I am not sure if I have writtin this tip anywhere? Your float value will be casted to an int automatically so there is no need to cast it before or to store it in a local register.
Your bug has been fixed. Thank you for reporting it. [Embed] is still broken though.
I also cleaned up the JAR so it is now less bloated. All my test files were included as well.
Hello Joa,
Looks like an astonishing tool you have here.
I try to get it running on a few of my SWF’s, but it keeps crashing for me. Here is the Java exception :
java.lang.RuntimeException: java.lang.NullPointerException
at com.joa_ebert.tools.tdsi.TDSI.run(TDSI.java:193)
at com.joa_ebert.tools.tdsi.TDSI.visit(TDSI.java:206)
at com.joa_ebert.swf.tags.control.DoABCTag.accept(DoABCTag.java:23)
at com.joa_ebert.swf.Swf.accept(Swf.java:69)
at com.joa_ebert.tools.tdsi.TDSI.run(TDSI.java:105)
at com.joa_ebert.tools.ToolRunner.run(ToolRunner.java:47)
at com.joa_ebert.tools.tdsi.TDSI.main(TDSI.java:36)
Caused by: java.lang.NullPointerException
at com.joa_ebert.abc.bytecode.MarkerManager.verifyMarker(MarkerManager.j
ava:255)
at com.joa_ebert.abc.bytecode.MarkerManager.applyPatches(MarkerManager.j
ava:50)
at com.joa_ebert.abc.bytecode.BytecodeEncoder.encode(BytecodeEncoder.jav
a:416)
at com.joa_ebert.abc.Abc.writeMethodBodies(Abc.java:1364)
at com.joa_ebert.abc.Abc.write(Abc.java:1065)
at com.joa_ebert.abc.Abc.write(Abc.java:1015)
at com.joa_ebert.tools.tdsi.TDSI.run(TDSI.java:189)
… 6 more
Any idea ?
Are you going to publish the source code of the tool BTW ?
Keep up the good work, it looks VERY promising !
I have released an update which will not fix your problem but creates a better exception.
It would be perfect if you could send me the SWF causing this exception.
Source will not be released before FOTB.
Hello again Joa,
Just tried the update, here is the result. I can’t really send you the SWF for some reasons, please email me (romain AT dotemu DOT com) to see what I can do. It’s a small SWF (47 Kb) but still a very complex program, so I might try to isolate the problem.
Here is the new detailed exception :
java -jar tdsi.jar -dead-code-elimination false -input MyTest.swf -output a.swf
[i] Running tool TurboDieselSportInjection …
[i] TDSI configuration:
[i] inlineBytecode = true
[i] integerCalculus = false
[i] bytecodePermutations = true
[i] inlineMemory = true
[i] deadCodeElimination = false
[-] com.joa_ebert.abc.bytecode.MarkerException: Marker got corrupted. Operation
is null.
java.lang.RuntimeException: com.joa_ebert.abc.bytecode.MarkerException: Marker g
ot corrupted. Operation is null.
at com.joa_ebert.tools.tdsi.TDSI.run(TDSI.java:193)
at com.joa_ebert.tools.tdsi.TDSI.visit(TDSI.java:206)
at com.joa_ebert.swf.tags.control.DoABCTag.accept(DoABCTag.java:23)
at com.joa_ebert.swf.Swf.accept(Swf.java:69)
at com.joa_ebert.tools.tdsi.TDSI.run(TDSI.java:105)
at com.joa_ebert.tools.ToolRunner.run(ToolRunner.java:47)
at com.joa_ebert.tools.tdsi.TDSI.main(TDSI.java:36)
Caused by: com.joa_ebert.abc.bytecode.MarkerException: Marker got corrupted. Ope
ration is null.
at com.joa_ebert.abc.bytecode.MarkerManager.verifyMarker(MarkerManager.j
ava:258)
at com.joa_ebert.abc.bytecode.MarkerManager.applyPatches(MarkerManager.j
ava:50)
at com.joa_ebert.abc.bytecode.BytecodeEncoder.encode(BytecodeEncoder.jav
a:416)
at com.joa_ebert.abc.Abc.writeMethodBodies(Abc.java:1364)
at com.joa_ebert.abc.Abc.write(Abc.java:1065)
at com.joa_ebert.abc.Abc.write(Abc.java:1015)
at com.joa_ebert.tools.tdsi.TDSI.run(TDSI.java:189)
… 6 more
I think I got the precise block where the bug occurs…
Here is the symbolic corresponding code :
public final function load(type:int, name:String):MyResource
{
switch (type)
{
case MyResource.TYPE_1: return loadType1(name);
default: trace(”Unsupported resource type for (”+name+”)”); return null;
}
return null;
}
If I comment the “default” case in the switch/case statement, it’s okay.
If I comment ONLY the “return null” statement in the default case, leaving the trace alone, it’s okay.
The full line make TDSI crashes badly, as stated in my previous messages.
So my guess is that you do not handle a “return null” properly in a default case from a switch/case statement (lookupswitch in ABC bytecode if I’m right).
Hope it helps !
Thank you very much for the example!
There are two possible problems:
- There is a strange problem for Jump operations that are dead code but jump outside of the method which the compiler produces in very rare cases. Those jumps are actually broken but for reading I made non-strict option that will ignore those broken jumps. This mode is not enabled when writing, which is already a problem.
- The labeled statement got removed for some reason but in those cases I can fold labels. But nothing should happen in your method since you do not use the Memory class or anything else.
Update: If you enable dead-code-elimination this problem should not occur. Anyways I have to figure out what is wrong.
Got it working by a workaround (getting the “return null” statement) out of the switch/case.
Well, it didn’t went very far, because the Flash Player stopped me when I tried to run the application with a
VerifyError: Error #1045: Unsupported traits kind=
And I’m not using any embed, this is just plain AS3 (tricky) code….
I’m willing to help on debugging it, mail me to see what we can do.
As I thought: your method fails because the ASC inserts a stupid Jump statement which I reject since it jumps to nowhere at 0×000019. Dead-code-elimination will remove this statement since it follows a ReturnValue.
The VerifyError is a problem but not a surprise. I think my AbcInputStream or AbcOutputStream class has a problem. To be honest: I never tested certain datatypes.
0x000000 GetLocal0 0x000001 PushScope 0x000002 Jump L0 0x000006 L1: Label 0x000007 PushString "type1" 0x000009 ReturnValue 0x00000a L2: Label 0x00000b FindPropStrict QName(PackageNamespace(""), "trace") 0x00000d PushString "Unsupported resource type for (" 0x00000f GetLocal2 0x000010 Add 0x000011 PushString ")" 0x000013 Add 0x000014 CallPropVoid QName(PackageNamespace(""), "trace"), 1 0x000017 PushNull 0x000018 ReturnValue 0x000019 Jump L3 0x00001d L0: GetLocal1 0x00001e SetLocal3 0x00001f PushByte 0x0 0x000021 GetLocal3 0x000022 IfStrictNotEqual L4 0x000026 PushByte 0x0 0x000028 Jump L5 0x00002c L4: Jump L6 0x000030 PushByte 0x1 0x000032 Jump L5 0x000036 L6: PushByte 0x1 0x000038 L5: Kill 3 0x00003a LookupSwitch L2, L1 L2Thank you for the report. I could successfully fix it. Strict mode is now disabled when writing and those jumps will not cause any trouble.
Now I will have to write some unit tests for the data types to get around the VerifyError problem.
Interesting dump :)
I could try the same way to isolate the problem for the VerifyError if you are interested in, just tell me.
The VerifyError is unfortunately much harder to address. I will have to go byte-by-byte through the output to see where everything fails.
Actually it is even worse because my output will probably never be the same since I rebuild the constant pool completly. But I do appreciate your interest in helping me very much!
In order to write tests more comfortable I wrote another utility for easier access to Abc data. This is more abstract and now used in TDSI as well so we have SWF and SWC support.
Short note: This is an evil problem and hard to debug. I already figured out that some bytes I write different have no impact and are even more “correct”.
I will continue searching for the problem tomorrow.
Ok Joa, keep us informed of the progress :)
The bug is finally fixed and an updated version is available. [Embed] works and your file might work too. Another problem still exists and I have to figure that one out. It may be the same problem.
[worship icon]
Thanks man! Now lets see what my Doom port does with the Memory class :-)
Hi,
Still no luck for me. I have tried the updated version playing with all options possible but :
1. Dead Code elimination = true makes TDSI hands during the process
2. Generated SWFs make the Debug Flash Player badly crash without raising any exception, and just don’t work in the release one. I tried on both a debug and release build of my sofware, same results.
Any idea ?
Hi Joa,
I’m really excited to get this going, but I’ve been encountering this error while trying to run your tool:
admins-macbook-pro:tdsi thomassaunders$ java -jar tdsi.jar -input MemoryTest.swf -output NewMemoryTest.swf
[i] Running tool TurboDieselSportInjection …
[i] TDSI configuration:
[i] inlineBytecode = true
[i] integerCalculus = false
[i] bytecodePermutations = true
[i] inlineMemory = true
[i] deadCodeElimination = true
Exception in thread “main” java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedList.addBefore(LinkedList.java:778)
at java.util.LinkedList.add(LinkedList.java:198)
at com.joa_ebert.abc.controlflow.Path.add(Path.java:20)
at com.joa_ebert.abc.controlflow.Path.clone(Path.java:45)
at com.joa_ebert.abc.controlflow.utils.PathBuilder.walk(PathBuilder.java:72)
at com.joa_ebert.abc.controlflow.utils.PathBuilder.walk(PathBuilder.java:61)
at com.joa_ebert.abc.controlflow.utils.PathBuilder.walk(PathBuilder.java:61)
at com.joa_ebert.abc.controlflow.utils.PathBuilder.walk(PathBuilder.java:61)
at com.joa_ebert.abc.controlflow.utils.PathBuilder.walk(PathBuilder.java:72)
at com.joa_ebert.abc.controlflow.utils.PathBuilder.walk(PathBuilder.java:61)
at com.joa_ebert.abc.controlflow.utils.PathBuilder.walk(PathBuilder.java:61)
at com.joa_ebert.abc.controlflow.utils.PathBuilder.walk(PathBuilder.java:61)
at com.joa_ebert.abc.controlflow.utils.PathBuilder.walk(PathBuilder.java:72)
at com.joa_ebert.abc.controlflow.utils.PathBuilder.walk(PathBuilder.java:61)
etc, etc..
Any ideas as to what is going wrong?
Yes. It is either an error in TDSI that is causing an endless loop for PathBuilder.walk() so you run out of memory or you have to increase the memory amount for Java.
Here is an example which allows TDSI to consume more memory:
java -Xms64m -Xmx384m -jar tdsi.jar -input input.swf -output output.swfIf it keeps crashing it would be very kind to send me your SWF.
Updated and fixed.
LTronic: Hopefully it will not crash for you. That was pure stupidity from my side as I introduced a bug while fixing yours.
Thomas: Thank you very much for your help. The error has been repleaced with a significant speed increase for bytecode analysis ;)
Hello Joa,
Just downloaded and tested the new version. TDSI runs fine without dead code elimination (turning it on still makes TDSI hangs).
I have now the following exception in FP10 :
TypeError: Error #1034: : Type Coercion failed: cannot convert Vector.@280ab81 in __AS3__.vec.Vector.
at CRC32/buildCRCTable()
Here is some sample code to reproduce it (well, I hope) :
public class CRC32
{
private static var CRCTable:Vector. = null ; // CRC Lookup table
private var crc:int; // currently calculated crc (used in conversion to byte array)
public function CRC32()
{
if (CRCTable==null)
buildCRCTable();
}
private function buildCRCTable():void
{
var CRC32_POLYNOMIAL:int = 0xEDB88320;
var i:int;
var j:int;
var crc:int;
CRCTable = new Vector.(256,true);
for (i = 0; i 0; j–)
if ((crc & 1) == 1)
crc = (crc >>> 1) ^ CRC32_POLYNOMIAL;
else
crc >>>= 1;
CRCTable[i] = crc;
}
}
}
Vector int (copy/paste didn’t very well)
Thanks. “Vector.@280ab81″ actually looks like some Java class got converted to string.
LTronic: I can not reproduce the bug with your code. Is it possible to send me the SWF?
Haha, dude, i seriously love the name. Finally a name that totally rocks! It embodies moviestar status compared to other frameworks.
Hi Joa,
Thanks for this, it’s exactly what I’ve been needing!
How can I run TDSI on a SWC? Only a small part of my application needs to be use the Alchemy opcodes, and running it on the complete SWF takes forever.
Oops, disregard my previous post. I see that the open-source version includes SWC support.
Unfortunately, now I get a NPE:
Caused by: java.lang.NullPointerException
at com.joa_ebert.apparat.abc.Metadata.equals(Metadata.java:49)
at com.joa_ebert.apparat.abc.Abc.getIndex(Abc.java:195)
at com.joa_ebert.apparat.abc.Abc.writeTraits(Abc.java:1657)
at com.joa_ebert.apparat.abc.Abc.writeInstances(Abc.java:1336)
at com.joa_ebert.apparat.abc.Abc.write(Abc.java:1089)
at com.joa_ebert.apparat.abc.Abc.write(Abc.java:1042)
at com.joa_ebert.apparat.tools.tdsi.TDSI.run(TDSI.java:234)
… 3 more
Seems the Metadata.name is null.
Thanks, fixed and uploaded. Just grab the latest version at http://www.joa-ebert.com/files/zip/tdsi.zip
I think you had actually a pretty old version of TDSI on your computer. Is that possible? The error should not occur under any circumstances now, but it does not make sense that it happened at all in the latest version. But I also do not know if I uploaded that version here in the blog :)
You’re right; I had some arbitrary old revision from SVN. Subsequent revisions fixed the particular problems I was having so I am able to process my SWF without any exceptions.
Unfortunately, when I try to run the resulting SWF, one of the methods with inlined Memory calls throws “VerifyError: Error #1024: Stack underflow occurred.” I can post more details in the Apparat discussion group.
Oh yes, please do so. That would be a great help!
Hello, Joa!
You are doing a great thing!
I tried to make some optimizations for loops, but I began to have problems with offsets for the IF-ops. Strange thing – if you consistently put the same bytes, for example, 0×02, 0×02, 0×02 (NOP), in the end we obtain NOP, DUP, DUP.
Is this a mistake?
If not, how to specify the address offset?
Great work Joa! Yet again a master piece!
Will be testing it these weeks. Will try to get you somke feedback.
Keep up the great work!
Anton,
what are you trying to do? You are using Apparat to optimize some loops, or the __bytecode method?
If you use Apparat, then the MarkerManager is your friend. The offsets for any jump instruction (Jump, If*, LookupSwitch) are converted into a Marker. Markers are solved in the BytecodeEncoder after the whole bytecode has been written. The distances which are datatype S24 are written with 0 values in the first pass. Once the exact offsets are known they are replaced with the actual value.
Hope this helps.
By the way, if you are interested in loop optimizations, the TAAS package is more interesting :o) I just have to finish some basic things and then we can have all optimizations we want. I just wrote a FlowOptimizer that does a great job in a couple of minutes.
joa,
I used pure __bytecode. I can’t understand, how to use Apparat.
I checked it out, I tried to compile with JDT, but there are only tdsi, dump and reducer as a result. Can you provide me an example about using jump instructions and/or TAAS?
Thank you very much!
Anton,
using __bytecode is nearly impossible. You need a method like __asm which I developed years ago for AS3C (as3c.googlecode.com). I will put such a method into Apparat as well.
TAAS is experimental but yields already great results. The implementation is at the moment simply incomplete.
It would be possible to write an __asm method in a couple of hours IMHO. Maybe I will do that in the next days but I can not promise it since my main focus is now getting TAAS done.
joa,
No problem! There are many interesting optimizations with Memory to do. :)
Thank you for your work!
What does this mean for the average Flash Developer (for who command line is a magical art)?
Can I just run it on my final SWF and the output will run faster?
Can I run it on Flash Player 9 files since it use the Alchemy bytecode and Alchemy was introduced with Flash Player 10?
Will we have to use versions of libraries optimised for TDSI like a version of as3corelib that does JPG encoding optimised to run TDSI after?
Wow that’s a lot of questions.
Keep up the good work
Thank you
zedia.net:
1) Google for command line.
2) No. In fact, you have to use the Memory API to make use of the Alchemy opcodes
3) I am not sure at which exact version the Alchemy opcodes have been introduced.
4) Sure. Someone would just have to write those libraries :)
Best,
Joa
Hi Joa,
Awesome work! We’re having some problems with writeShort / readUnsignedShort. Without tdsi the code runs fine (we are serializing a Vector. to a bytearray). After running tdsi we get stack underflow errors, any idea on these specific methods? (readFloat, writeFloat is fine..)
I also see that current operations are little endian on my intel machine. Is there a way to force endiannes? (we need this for serializing) And is this code safe on an older PPC mac?
Cheers!
Alexander
A tip for usage in a Flex environment: The Flex optimizer crashes on the Alchemy opcodes. So if you put a TDSI treated swf in a swc and compile that with mxmlc, there’s a good chance the whole thing won’t compile. I’ll file a bug report on the Adobe web site to that extent.