Continuous Deployment

Much has already been said about Continuous Deployment. Etsy made it famous once and we integrated our own solution about two weeks ago.

The reason why I integrated it into our system was quite simple: the only person who could perform a deploy was me. This was hindering the rest of the team of course.

Our architecture allowed “hot” deployments already. That means users who were already using the Audiotool application did not notice a deploy happened at all. This is of course only possible as long as there are no new dependencies on the API.

The Audiotool startup process is a very important ingridient. We have a boot sequence which loads a configuration file upfront. In this configuration file is a version number. The version number is used to load all dependencies the application needs to start. We have put a repository server in place which serves SWF files based on this version and they get cached till the end of time.

If a user started Audiotool when version 1.1 was online and we update to 1.2 in the meantime the user would still load all audio plugins for version 1.1. There are some cases when a hot deployment is not possible (yet) and we call this a scheduled update.
This pushes a message into Audiotool, notifying users that they should save their song and restart the application. But this is not part of the blog post and a scheduled update is very rare. We did it twice last year, once this year.

When we did a deploy it was basically done like this:

  1. Make sure all changes are checked in.
  2. Update a default version key in some source files in case the boot sequence cannot load the configuration for whatever reason.
  3. Update the Nginx configuration so that some verison-less files like our embed player are routed correct.
  4. Create a tag for the repository.
  5. Create a clone of the tag.
  6. Execute mvn -Pdeploy-to-live deploy in the clone. This command already updates all plug-ins on and puts new metadata in place.
  7. Copy all SWF files form a local directory to S3.
  8. SSH into a server and update the configuration file Audiotool parses at startup.
  9. Reload the Nginx configuration.

A lot of steps. This means a lot can go wrong of course. And even though we had all those steps written down in an internal Wiki it is still hard to do all this. You need the appropriate SSH keys to log into some servers, uploading files to S3 requires a tool which is also configured and not everyone is comfortable with editing a Nginx configuration file on an Amazon server through the terminal.

There was only one deployment that happened without me. I was at FITC Amsterdam in 2010 at that time and it was a long phone call. Obviously something had to change.

The first step was to get rid of all the manual configuration hassle. The configuration file that one needed to change via SSH had to go so I made it a dynamically created file by the web server. The actual version is pulled from the database and this made our my life much easier already. But still this was not what I wanted. Of course nothing really changed. You still had to perform the S3 upload, SSH into a server and so on.

But since the configuration was already served via the web server I could start automating more things. Even better: with our last major update we dropped the version number from Audiotool. User’s would stop expecting to see big changes and a version number that increases. Instead we focus on being much more active and to push changes online as quick as possible.

Since I already wrote some shell scripts for myself to deploy various server applications with a single command I started doing the same for the whole Audiotool application.

The first step was to get rid of the default version in some of the ActionScript source files. I simply [Embed] a text file now which contains the version information. A single text file is so much easier to change.

Then I added an API call which allows me to change the version information online. That way a new version can be released easily without any human interaction. With the API call in place and a text file containing all important configuration parameters the last issue was the Nginx configuration. But a small script on the server should do the job.

So I started writing a little script which performs all the necessary tasks:

  1. Create a clone of the repository.
  2. Get the id of the current revision.
  3. Create a tag for the revision like YYYY-MM-DD_REVISION.
  4. Replace the default version with the revision.
  5. Execute the Maven command to deploy some metadata and build all SWF files.
  6. Upload all SWF files to S3.
  7. curl our API with the revision information.
  8. SSH into a server at Amazon to update Nginx.
  9. Cleanup.

That’s all there was to it. Since I do not have that much experience with shell scripting the most annoying part was to figure out a way to replace the text in a file. After looking through all examples of possible/impossible sed and awk options I got lucky with sed -i "/@build.version@/ s//${HG_VERSION}/g" ${HG_CLONE}/default.version.txt. This basically replaces @build.version@ with the content of ${HG_VERSION}. I know, magic. I just want to write it down here so Future-Joa can Google this and come back in five years getting a quick answer.

With the shell script at our disposal we simply had to hook it into TeamCity, a fantastic continuous integration solution by the way. After configuring some command line tools on the CI server we were ready to go.

The results speak for themselves. We made already twenty deployments during the last two weeks. That is about as many deployments as we did since we started Audiotool. Because deployments become less scary and everyone can trigger them we are able to iterate quicker and get rid of a lots of bugs. In fact it makes it also much easier to reason about your program. If you push hundreds of new features and get a null pointer exception: good luck. However if you just changed one little thing and get loads of bug reports it is quite easy to identify what could be the possible culprit. Of course this is only a Flash problem since there are no stack traces in the release player. But I guess if you build a JavaScript application you would have similar problems.

I can only recommend doing the same. Especially for your own sanity. 100% automated deployments are really cool and stress free! It is also much easier to setup than you might expect.


  1. Subb
    Posted May 11, 2012 at 7:35 pm | Permalink

    I think you should rethink this new two columns layout, because it’s hard to read.

  2. Sebastien
    Posted May 11, 2012 at 7:38 pm | Permalink

    @Subb I agree. I used to religiously read Joa’s blog, but I couldn’t make it past half this post because of the layout…

  3. Posted May 12, 2012 at 10:14 am | Permalink

    Subb, Sebastien: Thanks for the suggestion. I was also not sure about it.

  4. slm
    Posted May 18, 2012 at 11:29 am | Permalink

    I agree. The old theme was better.

  5. Ole Christian Langfjæran
    Posted May 19, 2012 at 8:47 pm | Permalink

    If I read the post correctly I think you can replace step 1-4 and include them in your step 5 mavenbuild by using maven-filter-plugin. The plugin will replace all maven specific properties such as ${project.version} in selected files/folders. Alternatively some compiler options like CONFIG::buildversion could work?

  6. Simon Richardson
    Posted Jun 8, 2012 at 2:09 pm | Permalink

    Is there any information about how this works with databases?

    What happens to the data that could have been placed in the new revision or if there had to be changes in tables etc. Do you perform migration between revisions, if so this could get quite complicated?

  7. Posted Jun 11, 2012 at 5:57 pm | Permalink

    Simon: Yes of course. Your application could perform the migration during startup. All non-destructive (necessary) changes could be performed automatically. Sometimes if you roll back you do not have to alter the schema at all.

  8. Posted Sep 19, 2012 at 2:13 pm | Permalink

    This is awsome. I am totally impressed with it, and I must say I am not that into those kinda of things, in general.