If you are familiar with the concept of Continuous Deployment championed by Eric Ries of the Lean Startup fame, as well the motivations behind Immutable Infrastructure, you know how beneficial it is to pursue mechanisms enabling the fast and automated deployment/distribution of your application, the main ones being that the former decreases the feedback loop on new features while the latter removes setup/environment issues inherent with mutability in your environment.
There are plenty of utilities and frameworks to help us achieve that. Vagrant for example can be used to set up VMs. Obviously most cloud providers such as AWS, Azure, and DigitalOcean provide SDKs and APIs to let you automatically create VMs…but you would then need to build your own solution that would leverage them. Configuration Management tools like Ansible, Chef and Puppet can be leveraged to configure the newly created VMs.
When I was looking up how to do continuous deployment and build immutable infrastructure to deploy a web application, I came to realize I needed a way to set up my entire infrastructure on the cloud upon succesful completion of a CI build. I needed their configuration to vary based on parameters that I would define so as to reduce my costs between my UAT environment which would be built from my develop branch, and the Production environment which would be built from my master branch. I also needed a way for some of the resources I would have to create (namely my load balancers) to know the ip addresses of those VM instances. That solution should also provide a way for me to destroy that entire infrastructure. For the deployment of application, I needed it not only be immutable, but to also give me the necessary guarentees that it would work exactly as I would have tested it in my CI, and my UAT environment.
The solution that worked for me: Terraform for setting up all my VMs and Docker to package my application.
There’s plenty of information available on Docker, so I will not discuss it at length in this post. The simplest explanation I can give you on this is that docker is way to build application in layered images that can be run in a container. Once started, the container is “immutable”, i.e while you can change things on the VM it is running on (which would go against having Immutable Infrastructure), to my knowledge you cannot change or update the container itself. You can stop it, start it, restart it, attach to it, but you cannot change it…and that’s a very good thing.
Terraform.io is a utility written by the guys at Hashicorp (the company that built Vagreant). Its purpose is to provide means to code your infrastructure, set it all up and destroy it. It is built on Go yet has a rubyish feel to it. It enables the definition of variables and modules, and also comes bundled with quite a few provisioners for setting up resources on DigitalOcean and AWS to name a few. One of its greatest strength to me is the ability to reference resources and their properties in your terraform code. The output of terraform is also JSON parsable, which also opens up a host of possibilities.
One thing to keep in mind: Terraform is not meant to replace utilities like Chef or Puppet. Quite the contrary, you can still leverage them to configure your VM.
Say that like me you needed to setup your terraform environment upon succesfull completion of your build.
First, let’s define our variables files.
Now let’s create our module (NOTE: you don’t need to have a module. I am including this nonetheless for teaching purposes)
The above file should be placed in a “app_servers” folder. Please note that I am using CoreOS AMIs as they come prepackaged with Docker. Plus, they tend to take less time to boot up compared to Ubuntu AMIs.
And now, the final piece putting it all together.
To run it:
Simple isn’t it? For my actual code, I have a ruby script that loads up the appropriate values in my terraform variables file based on whether I am deploying from develop vs master, and backs up the resulting terraform outputs to AWS S3 in a versioned folder and sends me a HipChat message to tell me whether or not it was all successful. This enables me to destroy prior versions of my infrastructure upon successful creation of the new version, and it also helps me downgrade versions more easily. The script also determines which VMs to create depending on my current branch. In production, as per my example, I use an ELB. I do not do that in UAT so as to reduce my costs.
Terraform is obviously not a silver bullet, but it is a great tool to have on your devops utility belt. As it continues to mature towards its in 1.0 release, I’m confident it will become even more awesome.