Rebuilding out-of-date cloud server with chef-mailserver

Many, many years ago, I created a Slicehost server on which I host this blog post, many of my other websites and my mailserver. I used to keep it up to date. But these past four years, I’ve been remiss about maintaining it, initially due to laziness and later due to the immensity of the challenge of upgrading out-of-date software on a cloud server when the disk image and OS/packages must be kept in sync, compounded by how hard I’ve worked on my day jobs these past few years. This challenge was further complicated by Ubuntu ending upgrade support for the version I’m running and by Slicehost getting bought by Rackspace. Slicehost used to provide detailed instructions for simultaneously updating one’s Ubuntu version and one’s Slicehost disk image. That no longer exists.

So I’ve decided I should build a new server and migrate my data and services one-by-one to the new server.

Building a new server sounds like a job for Vagrant and Chef, and I found a vagrant-rackspace Vagrant plugin and a chef-mailserver cookbook for building a mail server. I’ve decided to give these a try. I’ll record my efforts here in case they’re helpful to anyone.

I like the choices made for chef-mailserver (postfix, dovecot, amavisd, clamav, spamassassin, and postfixadmin because they’re almost identical to the decisions I made years ago for my mail server. So these are time-tested technologies, and I’m familiar with most of them. Unfortunately, my existing database is MySQL while chef-mailserver uses PostgreSQL. But, while that’s a migration headache, I prefer Postgres, so I can live with that.

On my Debian desktop, I discovered the latest version of Vagrant is available as a package, so I installed it. I then ran vagrant plugin install vagrant-rackspace and got a “Failed to build gem native extension” error. The problem was that I didn’t have the ruby1.9.1-dev package installed. (I use Rbenv and generally don’t like to install/use Ruby packages from package manager.)

With vagrant-rackspace (0.1.6) installed, I followed the instructions at https://github.com/mitchellh/vagrant-rackspace, first running vagrant box add dummy https://github.com/mitchellh/vagrant-rackspace/raw/master/dummy.box, then creating a Vagrant file, for which I had to find my API key in my Rackspace Cloud account. I also chose some non-default settings, like setting rackspace_region to :ord because that’s where my existing server is, which could make migrating data/apps simpler and cheaper. (I’m not sure whether transferring data across regions counts as free private bandwidth or non-free public bandwidth.)

…After long break for work, kids, and a basement flooded by a dead hot water heater…

After further investigation, I was lured to try Digital Ocean instead. Their prices are great. And there’s a very popular vagrant-digitalocean plugin.

I used Vagrant to create a Digital Ocean “droplet.” I first created an account at DigitalOcean.com and provided my credit card information. I then uploaded my desktop’s public SSH key to the “SSH Keys” tab and got my Client ID and API Key from the “API” tab.

I installed knife-digital_ocean with gem install knife-digital_ocean and vagrant-digitalocean with vagrant plugin install vagrant-digitalocean.

I then filled in the Vagrantfile and overrode some defaults. Normally, I install the latest Ubuntu LTS release, but since a new LTS release is right around the corner (14.04 LTS is scheduled for April 2014), I decided to start with the current hotness (13.10) and upgrade to the LTS soon.

I created my “droplet” with vagrant up --provider=digital_ocean. I saw the information Vagrant stored about my new machine in .vagrant/machines/default/digital_ocean.

Vagrant no longer provisions machines with Chef by default. I forgot to do so, so after installing the vagrant-omnibus plugin (vagrant plugin install vagrant-omnibus), I added config.omnibus.chef_version = :latest to my Vagrantfile and ran vagrant reload --provision. It installed Chef 11.10.4-1.ubuntu.13.04.

Here’s a list of useful Vagrant commands:

vagrant destroy - Destroys the droplet instance.
vagrant ssh - Logs into the droplet instance using the configured user account.
vagrant halt - Powers off the droplet instance.
vagrant provision - Runs the configured provisioners and rsyncs any specified config.vm.synced_folder.
vagrant reload - Reboots the droplet instance.
vagrant rebuild - Destroys the droplet instance and recreates it with the same IP address is was assigned to previously.
vagrant status - Outputs the status (active, off, not created) for the droplet instance.

Running vagrant status shows my box has been given the default name:

Current machine states:
default                   active (digital_ocean)

I then used knife to see my new machine. First, I created a .chef/knife.rb file with two lines:

knife[:digital_ocean_client_id] = 'yyyyyyyyy'
knife[:digital_ocean_api_key] = 'xxxxxxxxx'

Running knife digital_ocean droplet list showed:

ID       Name     Size  Region      IPv4            Image                       Status
1234567  default  1GB   New York 2  107.170.123.123  1505699 (Ubuntu 13.10 x64)  active

I then tried logging in with ssh root@107.170.123.123, and it worked. Yay.

To install upgrades (esp. security upgrades), I ran apt-get update followed by apt-get upgrade.

I now want to use Chef to provision the server. There are multiple options for doing so. I’ve previously used Berkshelf and initially decided to do so again here. I ran gem install berkshelf and created a Berksfile like the following:

site :opscode
cookbook 'build-essential'

before realizing Berkshelf 2 (which is what gem install berkshelf installs) is now deprecated in favor of Berkshelf 3. So I ran gem uninstall berkshelf and followed the Berkshelf 3 instructions. I also had to add vagrant-digitalocean to both the Gemfile and Vagrantfile and change site :opscode to source "https://api.berkshelf.com". After doing so, I ran bundle exec vagrant reload --provision and got the expected output (on my fourth attempt).

But I gave up on Berkshelf for this project because its documentation currently warns against using version 2 but doesn’t document version 3. This situation should improve. But I’ve decided to give up the comfort of Berkshelf because its API appears in flux.

I installed gems for working with Chef recipes:

gem install knife-solo
gem install knife-github-cookbooks

And then used them to install cookbooks:

knife cookbook site install build-essential
knife cookbook github install pehlert/chef-mailserver
knife cookbook site install database
knife cookbook site lighttpd
knife cookbook site install php  # Yuck!

Installing cookbooks with knife is wonderful because knife usually identifies dependencies and installs them too. Also, knife will auto-commit your cookbooks into Git for you.

Before the above commands worked, I hit a snag (ERROR: IOError: Cannot open or read /home/jimmy/Git/jimmy-chef-mailserver/cookbooks/build-essential/metadata.rb! and ERROR: Errno::ENOENT: No such file or directory - /home/jimmy/Git/jimmy-chef-mailserver/cookbooks) that I fixed by force-deleting branches with git br -D chef-vendor-build-essential and git br -D chef-vendor-mailserver per the solution recommended here.

When I added the cookbooks' recipes to my runlist in Vagrant and re-ran vagrant reload --provision, this line in one of the recipes choked:

hosts = search(:node, "*:*").map { |n| n["ipaddress"] }

because the recipe was written for Chef Server only, not Chef Solo, which I knew because it complained:

Chef::Exceptions::PrivateKeyMissing
-----------------------------------
I cannot read /etc/chef/client.pem, which you told me to use to sign requests!

Googling let me to chef-solo-search, which I installed with knife cookbook github install edelight/chef-solo-search, but then I started hitting problems because my setup was trying to look for a Chef Server.

Frustration led me to a notice of the official deprecation process of Vagrant Berkshelf because “With so many moving parts, Vagrant Berkshelf became incredibly fragile; even the most subtle change in a transitive dependency would break the plugin, cascading as a flurry of issues on GitHub.”

I think I’ll stop using Chef-Solo and Berkshelf through Vagrant and instead use Vagrant just to boot up a minimal machine and then handle provisioning outside Vagrant. The article above recommends Test Kitchen: “There exists a better solution for provisioning virtual machines with Chef – Test Kitchen. Test Kitchen still leverages the amazing resolving power of Berkshelf under the covers, so switching is easier than ever. By switching to Test Kitchen, you are also no longer tied to Vagrant. You can use Berkshelf with your favorite cloud provider or bare metal!”

Though I haven’t yet read it, How To Use the DigitalOcean Plugin for Knife to Manage Droplets in Chef looks useful.

…TO BE CONTINUED… …CHECK BACK FOR MORE….

Posted by James on Saturday, February 22, 2014