If you ask me if ActionScript is ready for the enterprise then this question is hard to answer. Every fairly huge codebase requires some maintenance. You will have to figure out a strategy to identify modules and separate them. You will also have to handle the dependencies between modules. Last but not least you have to compile those.
An important aspect is that when I am talking about modules I am not talking about “Flex Modules” but the more generic term for a part of a project.
First of all you have to make sure to fulfill some requirements. You can not assume that every developer is working on the same platform using the same IDE with the same settings. But you must assume that your project will not be built from an IDE. You must also assure that the dependencies between projects are up to date across different platforms and IDEs. Once you have figured out how to do that you might want to think about a strategy to compile your modules.
Today I want to explore the idea of parallel compilation across a cluster of machines to yield optimal compile speed for ActionScript projects. First of all, you can not assume that a developer can always keep all modules up to date. Depending on the dependency graph of your projects compilation migh take very long. For instance we have 103 modules at Hobnox. This means we have 103 different compile targets with different dependencies, each yielding its own SWF/SWC file. We also have a “standard library” that is included in nearly every project. Changing this library will require the developer to recompile all 103 modules to update them.
To illustrate this example I want to explain what we went through.
- We used FDT to develop our project. Since FDT is Eclipse based every module was represented as a separate project with dependencies to other projects. FDT however compiles your project only once you launch it. And it does not do this automatically for your dependency graph. As a result you have to manually compile 103 modules in the correct order. This would not be the problem if FDT’s live error highlightning and code analysis would not work on a SWC basis. But since the actual source-code between projects is not analyzed we had to launch all 103 modules to propagate a possible error or change. This was for us the point when we had to make a switch.
- Ant was our second idea to build our FDT projects automatically. However we had to keep module dependencies up to date in all the different Ant files because the FCSH requires that you include external libraries of a library also in a successor of another library. This means if A requires B and C requires A. Then C requires also B. So we had to keep 103 Ant targets up to date for each change.
- A custom build. We started working with a custom build tool I have developed in Scala. This build tool was smart enough to figure out which modules have to be recompiled without launching the ActionScript compiler. It also figured out which projects could be compiled in parallel since the ActionScript compiler is written in a way that IO is not the only bottleneck. We had an enormous improvement in compile time and usability for the developer since module dependencies where handled automatically.
- We switched from FDT to IntelliJ IDEA since it offered us more features that we needed. IntelliJ came also with a much better strategy to compile modules. Basically it had the feature we always missed for FDT to automatically include libraries and to compile modules in the correct order automatically. However the guys at JetBrains made a mistake by counting on FCSH. The short story is that FCSH is absolutely not capable of handling a large codebase. What happened to us was that when we started a build FCSH would take up as much memory as it could, getting much slower until it finally crashed. Then FCSH started from scratch, consumed the maximum amount of memory again and crashed. This happend until a project was finally compiled but took also more than 15 minutes for a single change we made. This was of course unacceptable. Fortunately JetBrains implemented a feature request from us into IntelliJ IDEA which comes close to our custom build. From now on projects are compiled in parallel without using FCSH and without asking launching the compiler if no change has been made. This way we are able to work again. However only if we compile just a short amount of all modules since the main bottleneck is still the compiler.
- A continuous integration server was our last option. This is a custom build server and we are using it until today. After each SVN commit we are building the required modules and their dependencies now automatically on a dedicated server. A change in the standard library takes about four minutes to complete now on a quad core. But the good thing is that we have set our main project to exclude a lot of libraries. The custom build takes care of that and we are loading now modules from the server. That way we can still compile local and include libraries we frequently change. Other libraries are loaded from the server and included at runtime.
So the question is if this is now the ultimate solution. A custom continuous integration server that takes more than 4 minutes to propagate a change after a SVN commit? I do not think so. And here comes Scala into the game again. There is a project available called Swarm. It makes use of Scala’s support for serializable delimited continuations. This means you can pause the state of your current program, send it to a different node in your cluster and continue with the program on that machine. And Swarm makes this task extremely simple. So for now a new research project is to take advantage of this feature. We can basically have one master server like Hudson or TeamCity and write a plugin for it that makes use of Scala Swarm. Swarm could be based on nodes in the office, normal computers and maybe a couple of dedicated servers to compile modules in parallel on multiple nodes. Each node would have to keep the VCS up to date. That way we should be able to compile projects much quicker and have a much better workflow in a large codebase. Besides that if you work with a CI server than you could also have the task of code analysis using FlexPMD for instance done on another machine as well.
I am really looking forward to this project when I have some time to work on it. My goal is to develop a plug-in for TeamCity that we will use internally first. I expect to reduce the time for a full rebuild from about 4min to something like 30sec. This would be a huge benefit. I also hope that we will be able to make that plug-in available to the public.
great stuff, and excellent presentation on FITC…
brgds
Very interesting stuff Joa! What Swarm is doing for you reminds me of the render walls I used back in the early CGI with Silicon Graphics machines, for rendering on a farm of computers. Just set up a shell script to dish out chunks of frames to 20 odd purple Indys and go back to work!
Come along 2009/2010 with Cloud computing and Deju!
I must check out IntelliJ IDEA for large projects.
Thanks for your thoughts and work!
Guys from JetBrains already added a multi-process compilation with mxmlc/compc.
http://blogs.jetbrains.com/idea/2010/02/speed-up-flex-compilation/
Very useful!
This is for sure one of the most interesting topic not discussed enought.
So pls keep posting this kind of post.
Is also interesting to know deeply about tests (unit/integration/system tests)
i really appreciate also the fact you mainly decide not to talk about flex only stuffs (rising flash based project to a greater dignity).. looks like ppl keep thinking that only flex projects can fit the CI systems while mxmlc and amxmlc can easily compile As3 project, not talking about the upcoming (even if still present in cs4) .xfl
ups for your work ;)
Hi Joa
We use Maven to do the dependency management for us. Works like a charm.
As for recompilation of hundreds of modules – Couldn’t you create a bunch of interfaces which replace the classes in core module and put the inplementation into a rsl? This way you would only have to recompile, if one of the interfaces changes.
Best,
Ralf.
A interesting series on getting modularization right can be found here: http://techdistrict.kirkk.com/2009/11/16/applied-modularity-part-1/
FWIW, my favorite CI server is Buildbot. If Hudson or TeamCity are a bottleneck in your build process you might want to take a look at this lightweight master/slave(s) solution.
TeamCity/Hudson/Buildbot/… are not the bottleneck. And most support a master/slave releationship. However I would like to specialize this a little bit further.
The ideal solution is IMHO to have multiple FCSH instances on multiple machines, each compiling always the same target. That way you have long-running processes which are great for HotSpot and you save JVM startup costs which grow with the number of modules. You will also not run out of heap space. If I think of 2sec startup time for mxmlc/compc then this sums up to ~3min wasted time in our case. I think that only a custom solution will give us an accceptable result. And that one could be plugged into any CI system of your choice.
Sounds as if you should have went with haXe instead form the beginning. The Flex compiler is awfully slow! I’m wondering if you actually still get to write some AS code with all these auxiliary matters to care about.
If you are not using it already, there is something called HellFire compiler for flex, maby that will help. Try to look at blog by the ex Adobe guy Edwin Wong stopcoding.wordpress.com
Great post, thanks!
I agree with Ralf! Maven + http://flexmojos.sonatype.org/ works like charm.
It leverages most of the dependency problems you describe using Ant.
@Ole Christian Langfjaeran:
I didn’t know about Maven and Flexmojos, it seems very interesting. Is that possible to use those tools with an existing project ? I read in the “get started” that we have to create a Flex project structure with a shell command so I’m not sure I can use it in an already existing project.
Instead of Maven I would suggest checking out Buildr.
True, in FDT 3 we don’t have a good solution yet for building/compiling projects that consist of multiple projects.
At Powerflasher we normaly use ANT for that, which is not the best solution as you mentioned in your post. But for projects with 5-10 modules it’s perfectly ok in my opinion.
In FDT 4 Milestone 4 we are completely rewriting our compile system, will support modules and so on. I’m sure we can leverage your great knowldege about building and compiling large scale projects when we reach that point.
Thx,
Michael
-FDT Team
Hi,
back to what vane already mentioned, has anyone tried HFCD yet? Would be curious to hear any experiences.
Cheers
Sven
HFCD works very well with Flash Builder. However you can write your own HFCD very easy if you do not want to compile with an IDE.
As part of the Anvil Flex open source project we put together a fairly complex build process for a large enterprise application. I started to detail it in the comments, but it become so long I have posted in as a separate blog post at http://grandcloud.com/speeding-up-compilation-of-large-flex-3-appli
Interesting post Ryan! I just rewrote some parts of our custom build to work better with TeamCity and to scale on a local grid. So we have now pre-tested commits and very fast response time which is really cool :)
have you also tried out apache ivy for dependency management? there is an ant task called buildlist which should outputs a list of modules sorted due the dependency graph. with subant the compilation of the modules could be triggered. unfortunately i coulnd´t get it running until now… has anyone experience with it?
No we never tried Ivy. Even “tsort” in a shell can sort your modules in correct order if you are looking for that. Or you can sort a Map[T, List[T]] with three lines of Scala.
Anyways. The “real” dependency hell starts at runtime, not at compiletime :)
SVN? What century are you living in? Git it man….
Flash? I’d be careful now also… the world is changing…