Since Voxel hosts some of the web’s most widely read, most highly trafficked websites, we run across (and thrive on) complex, multi-server, multi-stage deployments. This means that we’re continually trying to find the best way to automate releases into these environments. Recently, we were tasked with figuring out the best way to push code updates onto a large web cluster running a popular Ruby on Rails application. Here’s what we did:
Aside from being a being a pretty and colorful picture, there’s a lot happening here. In this example, we have five web/app servers that are running the Rails app (let’s call it “myapp”), two database servers setup in master/slave replication, one staging server, one testing server, and one SVN server. The load balancer distributes traffic evenly between the web/app servers.
On the web/app servers we are using mongrels to serve up HTTP to Apache, and using god to monitor mongrels and make sure our application is happy and responding. Think it’s going to be hard to push code updates to this cluster? Think it’s going to be hard to add more web/app servers and keep everything up to date? Think again. We can setup new web/app servers and update code on existing servers with one command:
cap production deploy
I’m getting ahead of myself. First, how do we setup the Capistrano server? Here’s a post-install script for you:
# make sure we can build gems yum install -y ruby ruby-libs ruby-devel ruby-rdoc libxml2 libxml2-devel gcc # install ruby gems wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz tar -zxvf rubygems-1.3.5.tgz cd rubygems-1.3.5 ruby setup.rb install # using gems, install everything we need gem install rails capistrano xml-simple libxml-ruby capistrano-ext # install san_juan which extends capistrano functions to god gem install jnewland-san_juan -s http://gems.github.com # install the hapi client for ruby wget http://api.voxel.net/clients/ruby/ruby-hAPI-1.0.5.tar.gz tar -zxvf ruby-hAPI-1.0.5.tar.gz cd ruby-hAPI sed -i "s/ 'ParaOpts' => '>= 1.0.0',//g" hapi.gemspec gem build hapi.gemspec gem install hapi # setup our capistrano application and capify it mkdir -p /var/local/www/ cd /var/local/www/ rails capistrano cd capistrano capify .
This will setup a new server with a new capistrano app in /var/local/www/capistrano/ (let’s call it {cap_dir} for short). This relies on the capistrano-ext gem which allows for multistage deployment. Now, setup your environments by changing your {cap_dir}/config/deploy.rb file to this:
set :stages, %w(testing staging production) require 'capistrano/ext/multistage'
By setting up multiple stages, this will allow you to execute commands in certain environments, like `cap staging deploy` or `cap production migrate`. The environment-specific capfiles are located in {cap_dir}/config/deploy/{environment}.rb. Let’s take a look at production.rb:
require 'san_juan' require 'hapi' # the environment, globally $environment = "production" # the application name set :application, "myapp" # where we're deploying to on the web/app servers, and set it globally set :deploy_to, "/var/local/www/myapp" $deploy_to = deploy_to # set hapi credentials so we can get an up-to-date list of servers we're deploying to # the preferred method of authentication is to use hAPI authkeys. more information # can be found here set :hapi_username, "hapiusername" set :hapi_password, "hapipassword" # where the repository is currently at set :repository, "svn://svn.mydomain.com/svn/myapp/branches/trunk" # how we'll be getting the code from the repository set :scm, :subversion # specify the credentials for the repository set :scm_username, "svnusername" set :scm_password, "svnpassword" # this ssh user needs to be able to login on every web/app/db server and have # permissions in the $deploy_to directory set :runner, "myapp_deployment" # don't use sudo (make sure the :runner user has $deploy_to directory permissions) set :use_sudo, false # set this server manually so the rake db:migrate gets run properly, and only once role :db, "app1.mydomain.com", :primary => true ################################################### # end configuration, start running ################################################### # use the remote copy and update instead of checking out the entire app every time set :deploy_via, :remote_cache # make sure we don't copy files we don't want or need set :copy_exclude, [".svn", ".DS_Store"] # authenticate with hapi hapi = HAPI.new( { :hapi_username => hapi_username, :hapi_password => hapi_password } ) # get all devices from hapi labels = hapi.voxel_devices_list.map { |d| d['label'] } # IMPORTANT: # this will assign the roles of app and web to any device whose label matches "app", like # "app1.mydomain.com" and "app24.mydomain.com". It's important that there aren't other # servers that match this regular expression, and it's also VERY important that the labels # are resolvable FQDNs because capistrano will connect via ssh (e.g. `ssh {$label}`). # in this case, the app servers act as the web servers as well. If they didn't, you could edit # this to regex on "web" (or whatever you'd standardized on) role(:app) { labels.select { |l| l =~ /app/ } } role(:web) { labels.select { |l| l =~ /app/ } } # specify which servers need the mongrels cluster stopped and started # IMPORTANT: this name (e.g. "mongrels") needs to match the w.group = 'mongrels' # in the config.god on the web/app servers. Click here for more # information. Search for "grouping watches" to see where and how to # specify this information. san_juan.role :app, %w(mongrels) ################################################### # end setup, start code ################################################### # before trying to update our app, make sure the app is actually setup on this server before "deploy:update" do # check to see if the releases directory exists run "if [ -d '#{$deploy_to}/releases' ]; then echo yes; else echo no; fi;" do |channel, stream, data| # if it's not setup if data.strip != "yes" then # deploy the app `cap #{$environment} deploy:setup` end end end # overload some methods to use god to stop and start mongrels on the web/app servers namespace :deploy do desc <<-DESC Start the mongrels cluster on the web/app server using god DESC task :start do god:app.mongrels.start end desc <<-DESC Stop the mongrels cluster on the web/app server using god DESC task :stop do god:app.mongrels.stop end desc <<-DESC Restart the mongrels cluster on the web/app server using god DESC task :restart do deploy:stop deploy:start end end
Issuing a `cap production deploy` from the Capistrano server will login to each of the web/app servers and update the code to the latest version and restart mongrels using god. For a detailed description of what happens when you call `deploy` see here. Make sure to setup ssh_keys for the “myapp_deployment” user on all servers so capistrano won’t prompt for a password. (The SSH keys tutorial I linked to leaves one thing out: The authorized_keys file needs to be chmod 0600 and the ~/.ssh/ directory needs to be chmod 0700.)
It might take an hour or so to set this up and test it properly, but it will save you countless hours in the long run. Instead of logging into each server and issuing a set of commands, you can sit back and issue one deploy command. This is especially helpful in multi-stage, multi-server environments.
Posted in General Posts | No Comments »
